Methods for automatically generating a common measurement across multiple assembly units

ABSTRACT

A method includes: displaying a first image of a first assembly unit within a user interface; locating a first virtual origin at a first feature on the first assembly unit; displaying a first subregion of the first image within the user interface responsive to a change in a view window of the first image; recording a geometry and a position of the first subregion relative to the first virtual origin; locating a second virtual origin at a second feature—analogous to the first feature—on a second assembly unit represented in the second image; projecting the geometry and the position of the first subregion onto the second image according to the second virtual origin to define a second subregion of the second image; and, in response to receipt of a command to advance from the first image to the second image, displaying the second subregion within the user interface.

CROSS-REFERENCE TO RELATED APPLICATIONS

This Application claims the benefit of U.S. Provisional Application No.62/279,174, filed on 15 Jan. 2016, which is incorporated in its entiretyby this reference.

TECHNICAL FIELD

This invention relates generally to the field of optical inspection andmore specifically to new and useful methods for automatically generatinga common measurement across multiple assembly units in the field ofoptical inspection.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a flowchart representation of a first method;

FIG. 2 is a graphical representation of one variation of the firstmethod;

FIG. 3 is flowchart representations of a second method;

FIG. 4 is flowchart representations of one variation of the secondmethod;

FIG. 5 is a graphical representation of one variation of the secondmethod;

FIG. 6 is a flowchart representation of a third method;

FIG. 7 is a graphical representation of one variation of the thirdmethod;

FIG. 8 is a graphical representation of one variation of the thirdmethod; and

FIGS. 9A, 9B, and 9C are graphical representations of variations of thethird method.

DESCRIPTION OF THE EMBODIMENTS

The following description of embodiments of the invention is notintended to limit the invention to these embodiments but rather toenable a person skilled in the art to make and use this invention.Variations, configurations, implementations, example implementations,and examples described herein are optional and are not exclusive to thevariations, configurations, implementations, example implementations,and examples they describe. The invention described herein can includeany and all permutations of these variations, configurations,implementations, example implementations, and examples.

1. Assembly Line Configuration

As shown in FIG. 1, a first method S100 for automatically configuringoptical inspection along an assembly line includes: retrieving a firstimage captured by a first optical inspection station at a first time inBlock S111, the first image associated with an identifier of the firstoptical inspection station and a first timestamp corresponding to thefirst time; retrieving a second image captured by a second opticalinspection station at a second time succeeding the first time in BlockS112, the second image associated with an identifier of the secondoptical inspection station and a second timestamp corresponding to thesecond time; retrieving a third image captured by the first opticalinspection station at a third time succeeding the first time in BlockS113, the third image associated with the identifier of the firstoptical inspection station and a third timestamp corresponding to thethird time; identifying a first serial number of a first assembly unitin the first image in Block S121; identifying the first serial number inthe second image in Block S122; identifying a second serial number of asecond assembly unit in the third image in Block S123; determiningpositions of the first optical inspection station and the second opticalinspection station along the assembly line based on the first timestamppreceding the second timestamp and identification of the first serialnumber in the first image and in the second image in Block S130;determining positions of the first assembly unit and the second assemblyunit along the assembly line at a particular time based onidentification of the first serial number in the second image associatedwith the second timestamp and identification of the second serial numberin the third image associated with the third timestamp in Block S140;within a user interface, rendering a virtual representation of theassembly line and virtual representations of the first assembly unit andthe second assembly unit along the assembly line at the particular timebased on the determined positions of the first optical inspectionstation and the second optical inspection station and the determinedpositions of the first assembly unit and the second assembly unit at theparticular time in Block S150.

1.1 Applications

Generally, a production validation system (hereinafter the “system”) canexecute Blocks of the first method S100 to automatically configuremultiple optical inspection stations along an assembly line after theassembly line and optical inspection stations have been installed andafter images of units passing through the assembly line have been imagedby the optical inspection stations. In particular, the first method S100can be executed by a local or remote computer system that communicateswith one or more optical inspection stations to collect images ofassembly units in near real-time, interfaces with a local or remotedatabase to retrieve stored images, and/or hosts a user interface (e.g.,at a user's smartphone, tablet, or desktop computer) to serve images andrelated data to the user and to receive image selections and otherinputs from the user. Optical inspection stations (described below) canbe inserted into an assembly line at various assembly stages andimmediately used to capture images of units passing through the assemblyline. The optical inspection stations can upload these images to a(local or remote) database, such as in real-time or asynchronously, witha timestamp of when each image was captured and an identifier of theoptical inspection station that captured each image (e.g., “imagemetadata”). The system can then execute Blocks of the first methodS100—such as locally at a computer system connected directly to theassembly line, within a native application or web browser executing on amobile computing device logged into the assembly, or remotely at aremote server—to automatically identify optical inspection stationsinserted into the assembly line, to automatically identify the order ofoptical inspection stations along the assembly line, and toautomatically determine the positions of various units along theassembly line at a particular (e.g., current) instance in time based onvisual data contained in the images and image metadata received from theoptical inspection stations. The system can then automatically configurea virtual representation of the assembly line, including relativepositions of optical inspection stations and relative positions of unitsalong the assembly line, as shown in FIG. 2. The system can thereforeexecute Blocks of the first method S100 to automatically configureoptical inspection stations along an assembly line and to generate avirtual representation of the state of units within the assembly linefor presentation to a user substantially in real-time as units areassembled.

The system can execute the first method S100 to collect, process, andmanipulate images of test assemblies (hereinafter “units”) duringproduct development, such as during a prototype build, an engineeringvalidation test (EVT), design validation test (DVT), and/or productionvalidation test (PVT). The system collects, processes, and manipulatesimages of units captured by one or more optical inspection stationsduring a prototype build event (or “build”), such as over several hours,days, or weeks in which dozens, hundreds, or thousands of units areassembled and tested. The system can also be implemented within a batchor mass production assembly line for in-process quality control, earlydefect detection, etc. within a production run. The system can alsointegrate into a manual-pass-type assembly line or into a conveyor-typeassembly line.

Furthermore, the system can be implemented across a distributed assemblyline, such as across physically co-located assembly lines or assemblylines installed in different buildings on a single campus, installed ondifferent campuses of the same company or different companies, and/orinstalled in different cities, regions, countries, or continents. Forexample, multiple sets of optical inspection stations can be installedwithin each of multiple discrete and remotely-located assembly lines fora product or subassembly of a product, and the system can aggregateimages captured and uploaded by the optical inspection stations into asingle distributed assembly line for the product or subassembly.Similarly, the system can be implemented within a raw materialprocessing facility, an injection molding facility, a casting andmachining facility, any sub-assembly or main assembly layer in assemblyfacility, an inspection facility, a testing and reliability testfacility, a validation or failure analysis facility, a packing facility,a shipping facility, a field use facility, a field return facility,and/or a field return failure analysis, etc. during production, testing,and/or validation of a single component, a subassembly, a main assembly,etc. The system is described herein for integration with an assemblyline. However, the system can be integrated with any one or moremanufacturing, assembly, testing, validation, and/or other productionprocesses for a single component, subassembly, main assembly, etc.(hereinafter a “unit”).

1.2 Optical Inspection Station

The system includes one or more optical inspection stations. Eachoptical inspection station can include: an imaging platform thatreceives a part or assembly; a visible light camera (e.g., a RGB CMOS,or black and white CCD camera) that captures images (e.g., digitalphotographic color images) of units placed on the imaging platform; anda data bus that offloads images, such as to a local or remote database.An optical inspection station can additionally or alternatively includemultiple visible light cameras, one or more infrared cameras, a laserdepth sensor, etc.

In one implementation, an optical inspection station also includes adepth camera, such as an infrared depth camera, configured to outputdepth images. In this implementation, the optical inspection station cantrigger both the visible light camera and the depth camera to capture acolor image and a depth image, respectively, of each unit set on theimaging platform. Alternatively, the optical inspection station caninclude optical fiducials arranged on and/or near the imaging platform.In this implementation, the optical inspection station (or a local orremote computer system interfacing with the remote database) canimplement machine vision techniques to identify these fiducials in acolor image captured by the visible light camera and to transform sizes,geometries (e.g., distortions from known geometries), and/or positionsof these fiducials within the color image into a depth map, into athree-dimensional color image, or into a three-dimensional measurementspace (described below) for the color image.

The system is described herein as including one or more opticalinspection stations and generating a virtual representation of anassembly line including the one or more optical inspection stations.However, the system can additionally or alternatively include any othertype of sensor-laden station, such as an oscilloscope station includingNC-controlled probes, a weighing station including a scale, a surfaceprofile station including an NC-controlled surface profile gauge, or astation including any other optical, acoustic, thermal, or other type ofcontact or non-contact sensor.

1.3 Automatic Configuration

Following insertion of a set of optical inspection stations into anassembly line, the optical stations can capture and upload color imagesof units passing through the optical inspection stations to a local orremote database. Upon receipt of an image from a deployed opticalinspection station, the system can: implement optical characterrecognition techniques or other machine vision techniques to identifyand read a serial number, barcode, quick-response (“QR”) code, or othervisual identifier of a unit within the image; generate an alphanumerictag representing this serial number, barcode, QR code, or other visualidentifier; and then add this alphanumeric tag to the metadata receivedwith the image. The system can thus receive images of various units inBlock S111, S112, and S113 and then read identifying information ofthese units from these images in Blocks S121, S122, and S123. (Eachoptical inspection station can alternatively include an RFID reader, anNFC reader, or other optical or radio reader that locally reads a serialnumber from a unit placed on its imaging platform, and the opticalinspection station can add a serial number read from a unit to metadataof an image of the assembly unit.)

In Block S130, the system can then process unit serial numbers, opticalinspection station identifiers (e.g., serial numbers), and timestamps(i.e., times that units of known unit serial numbers entered opticalinspection station of known identifiers) contained in metadata of imagesreceived from the optical inspection stations to determine the order ofoptical inspection stations along the assembly line, as shown in FIG. 1.In one implementation, as images are received from optical inspectionstations, the system: buckets a set of images containing a tag for aspecific unit serial number; extracts optical inspection station serialnumber tags and timestamps from metadata in this set of images; andorders these optical inspection station serial numbers (from first tolast in an assembly line) according to their corresponding timestamps(from oldest to newest). In particular, a unit progresses throughassembly over time and is sequentially imaged by optical inspectionstations along an assembly line, and the system can transform unitserial numbers, optical inspection station serial numbers, andtimestamps stored with images received from these optical inspectionstations into identification of a group of optical inspection stationsas corresponding to one assembly line and confirmation of the order ofthe optical inspection stations along this assembly line. The system canrepeat this process for other unit serial numbers—such as for eachserial number of a unit entering the first optical inspection station inthis ordered set of optical inspection stations—in order to confirm thedetermined order of optical inspection stations along an assembly lineand to automatically detect reconfiguration of optical inspectionstations on the assembly line (e.g., in real-time).

In this implementation, the system can also pass these opticalinspection station serial numbers into a name mapping system (e.g., aDNS) to retrieve optical inspection station-specific information, suchas make, model, last user-entered name, configuration (e.g., imagingplatform size, optical resolution, magnification capacity), owner orlessee, etc. for each optical inspection station. The system cansimilarly pass unit serial numbers into a name mapping system or otherdatabase to retrieve unit-specific data, such as assigned build,configuration, bill of materials, special assembly instructions,measurements, photographs, notes, etc.

In Block S150, the system can then generate a virtual representation ofthe ordered optical inspection stations along an assembly line, such asshown in FIG. 2. The system can label virtual representations of theoptical inspection stations with a make, model, name, configuration,serial number, etc. retrieved from a remote database or according to aname or description entered by a user. The system can then upload thevirtual representation of the assembly line to a local or remotecomputer system (e.g., a smartphone, a tablet, a desktop computer) foraccess by a user. The system can also receive images from opticalinspection stations across multiple distinct assembly lines and canimplement the foregoing methods and techniques substantially in realtime to bucket images of units on different assembly lines, to identifymultiple assembly lines and optical inspection station order in eachassembly line, to generate a unique virtual representation of eachassembly line represented by the images, and to distribute these virtualassembly line representations to their corresponding owners.

The system can also repeat the foregoing methods and techniquesthroughout operation of the assembly line in order to detect insertionof additional optical inspection stations into the assembly line, todetect removal of optical inspection stations from the assembly line,and/or to detect rearrangement of optical inspection stations within theassembly line and to automatically update the virtual representation ofthe assembly line accordingly.

1.4 Assembly Line Status

In Block S140, the system can identify the current position of a unitwithin the assembly line based on the optical inspection station serialnumber tag stored with the last image—containing a unit serial numberfor the assembly unit—received from the assembly line. For example, fora unit within the assembly line, the system can determine that aparticular unit is between a first optical inspection station and asecond optical inspection station along the assembly line if the lastimage containing a unit serial number tag for the particular unit wasreceived from the second optical inspection station (i.e., contains anoptical inspection station serial number tag for the first opticalinspection station). In this example, the system can determine that theparticular unit is at the second optical inspection station along theassembly line if the last image containing the assembly unit serialnumber tag for the particular unit was recently received from the secondoptical inspection station by another image of another unit that has notyet been received from the second optical inspection station.Furthermore, in this example, the system can determine that assembly ofthe particular unit has been completed if the last image containing theassembly unit serial number tag for the particular unit was receivedfrom a last known optical inspection station on the assembly line.

The system can repeat the foregoing process for unit serial numbers ofother units identified in images received from the optical inspectionstations inserted along an assembly line. The system can then populatethe virtual representation of the assembly line described above with aheat map of current unit position, as shown in FIGS. 1 and 2. As eachnew image is received from an optical inspection station on the assemblyline and a new position of a particular unit along the assembly linethus determined, the system can update the virtual representation of theassembly line to reflect the new determined position of the particularunit in Block S150. The system can also pulse or otherwise animate amarker representative of the particular unit within the virtualrepresentation of the assembly line to visually indicate to a user thatthe particular unit has moved.

The system can implement similar methods and techniques to generate aheat map or other virtual representation of the assembly line at aparticular previous time selected by a user based on last images ofunits received from optical inspection stations along the assembly lineprior to the particular previous time. The system can thereforerecalculate assembly line states and unit positions at previous timesand display virtual representations of these assembly line states forthe user substantially in real-time as a user scrolls through a timehistory of the assembly line. The system can also filter images receivedfrom optical inspection stations, such as by build, configuration, dateor time, inspection time, etc. based on a user selection for a subset ofunits on the assembly line; the system can then calculate an assemblyline state for the subset of units from the filtered images and displaya virtual representation of this assembly line state in Block S150.

However, the system can execute Blocks of the first method in any otherway to transform images received from optical inspection stations into aconfiguration of optical inspection stations along an assembly line andto determine a state of units along the assembly line.

1.5 Defect Detection

In one variation, the system implements machine vision techniques todetect manufacturing defects along the assembly line and augments thevirtual representation of the assembly line with locations, types,and/or frequencies of manufacturing defects detected in units passingthrough the assembly line. For example, the system can implement methodsand techniques described below to analyze images of units to detectfeatures (e.g., component dimensions, absolute or relative componentpositions) that fall outside of dimensions and tolerances specified forthe feature. In another example, the system can implementtemplate-matching techniques to detect scratches, dents, and otheraesthetic defects on units in images received from optical inspectionstations along the assembly line.

In this variation, when a defect on a unit is detected in an earliestimage of the assembly unit, the system can flag a unit serial numbercorresponding to the image in which the defect was detected and theninsert a defect flag into the virtual representation of the assembly ata particular optical inspection station at which this image wascaptured. The system can thus visually indicate to a user through thevirtual representation of the assembly line that the defect on theassembly unit occurred between the particular optical inspection stationand a second optical inspection station immediately preceding theparticular optical inspection station in the assembly line. Furthermore,if the system detects defects shown in multiple images captured at aparticular optical inspection station, the system can identify defectsof the same type (e.g., similar scratches in the same area on a housingacross multiple units) and incorporate a counter for defects of the samedefect type into the virtual representation of the assembly line. Thesystem can also visually represent frequency, type, and/or position ofdetected defects across a batch of units passing through one or moreoptical inspection stations, such as in the form of a heatmap Forexample, the system can generate or access a virtual representation of a“nominal” e.g., “generic”) unit, calculate a heatmap containing a visualrepresentation of aggregate defects detected in like units passingthrough a single optical inspection station or passing through multipleoptical inspection stations in the assembly line, and then present theheatmap overlayed on the virtual representation of the nominal unitwithin the user interface.

However, the system can implement any other method or technique toidentify a defect in a unit shown in an image captured by an opticalinspection station within an assembly line and to indicate the earliestdetected presence of this defect on the assembly unit in the virtualrepresentation of the assembly line.

2. Window Mapping

As shown in FIG. 3, the second method S200 for rendering images ofassembly units along an assembly line, including: rendering a firstimage of a first assembly unit captured at an optical inspection stationwithin a user interface in Block S210; selecting a first feature withinthe first image corresponding to the first assembly unit as an origin ofthe first image in Block S220; in response to receipt of a zoom input,rendering an expanded region of the first image in the user interface inBlock S212; storing a size and a position of the expanded regionrelative to the origin of the first image in Block S230. The secondmethod S200 also includes, in response to advancement from the firstimage to a second image of a second assembly unit captured at theoptical inspection station: selecting a second feature within the secondimage corresponding to the first feature in the first image as an originof the second image in Block S240; selecting an expanded region of thesecond image corresponding to the size and the position of the expandedregion of the first image relative to the origin of the second image inBlock S242; and rendering the expanded region of the second image in theuser interface in Block S250.

One variation of the second method S200 includes: displaying a firstimage of a first assembly unit within a user interface in Block S210, aform of the first image recorded at an optical inspection station;locating a first virtual origin at a first feature on the first assemblyunit represented in the first image in Block S220; in response to achange in a view area of the first image at the user interface,displaying a first subregion of the first image within the userinterface in Block S212; and recording a geometry and a position of thefirst subregion of the first image relative to the first virtual originin Block S230. The second method S200 also includes, in response toreceipt of a command to advance from the first image to a second imageof a second assembly unit at the user interface: locating a secondvirtual origin at a second feature on the second assembly unitrepresented in the second image in Block S240, the second feature on thesecond assembly unit analogous to the first feature on the firstassembly unit; projecting the geometry and the position of the firstsubregion of the first image onto the second image according to thesecond virtual origin to define a second subregion of the second imagein Block S242; and displaying the second subregion of the second imagewithin the user interface in Block S250.

As shown in FIG. 4, another variation of the second method S200includes: displaying a first image of a first assembly unit within auser interface in Block S210, the first image recorded at an opticalinspection station; in response to a change in a view window of thefirst image at the user interface, displaying a first subregion of thefirst image within the user interface in Block S212; recording ageometry and a position of the first subregion of the first imagerelative to a first feature represented in the first image in BlockS230; identifying a second feature represented in a second image of asecond assembly unit in Block S240, the second feature analogous to thefirst feature; projecting the geometry and the position of the firstsubregion of the first image onto the second image according to thesecond feature to define a second subregion of the second image in BlockS242; and in response to receipt of a command to advance from the firstimage to the second image at the user interface, displaying the secondsubregion of the second image within the user interface in replacementof the first image in Block S250.

2.1 Applications

Blocks of the second method S200 can be executed locally by a computingdevice (e.g., a smartphone, a tablet, a desktop computer) incommunication with the remote computer system to display images of a setof assembly units (or “units”) captured by an optical inspection stationalong an assembly line. During an image viewing session, the computingdevice can display images of units of one or more assemblyconfigurations across one or more builds within a user interfacerendered on a display integrated into or connected to the computingdevice (e.g., within a native application or web browser executing on asmartphone, a tablet, or a desktop computer.)

Generally, when a user scrolls through a set of images of unitspreviously captured by a particular optical inspection station, thecomputing device can execute Blocks of the second method S200 toautomatically apply a last zoom level and viewing position from oneimage rendered in the user interface to a next image selected by theuser, thereby enabling the user to scroll through the same expanded viewacross a sequence of images of multiple units recorded at the same pointalong an assembly line. In particular, when the user scrolls from afirst image to a second image in a set of images of like units, thecomputing device (or a desktop or handheld computing device interfacingwith the computing device) can execute Blocks of the second method S200:to map a last viewing window from a first image of a first unit to asecond image of a similar second unit at the same assembly stage basedon features within the first and second images; and to automaticallyrender an expanded view of the second image—corresponding to the lastrendered expanded view of the first image—in the user interface suchthat the user can quickly and visually compare local differences betweenthe first unit and the second unit relative to a common virtual origin.

In one example, the computing device can aggregate a set of images of aPCB assembly—including a PCB and components attached to the PCB—capturedby a particular optical inspection station during a build. When a userreviews the first image in a set during a first image review session,the computing device can: implement machine vision techniques to detectan upper left corner and an upper horizontal edge of a first PCB shownin the first image; set the upper left corner of the first PCB as thevirtual origin in the first image; align the X-axis of the virtualorigin in the first image to the upper horizontal edge of the first PCB;render a region of the first image in the user interface; store the Xand Y coordinates of the upper-left corner pixel, the upper-right cornerpixel, the lower-left corner pixel, and the lower-right corner pixel ofthe first image rendered in the user interface, relative to the virtualorigin, for the last zoom level and viewing position for the firstimage. When the user then scrolls to a subsequent image in the set, thecomputing device can similarly: implement machine vision techniques todetect an upper left corner and an upper horizontal edge of a second PCBshown in the second image; set the upper left corner of the second PCBas the virtual origin in the second image; align the X-axis of thevirtual origin in the second image to the upper horizontal edge of thesecond PCB; and immediately render a rectangular region of the secondimage bounded by four corner pixels at the corner pixel coordinatesstored at the last viewing position of the first image. In this example,because the first image and the second image are (virtually)translationally aligned by like virtual origins and rotationally alignedby like X-axis reference features, and because the relative position ofthe first region of the second image rendered in the user interface—oncethe user scrolls to the second image—is substantially identical to therelative position of the last region of the first image rendered in theuser interface, the user can visually detect differences betweencomponent placement in the first PCB assembly shown in the first imageand component placement in the second PCB assembly shown in the secondimage. The computing device can repeat this process when the userscrolls from the second image back to the first image or when the userscrolls from the second image to a third image in the set. As shown inFIG. 5, the computing device can also implement Blocks of the secondmethod S200 to render like regions of images of like unitssimultaneously, such as in a grid layout or aligned by the view windowand laid over one another.

Blocks of the second method S200 are described below as executed by a“system.” For example, Blocks of the second method S200 can beimplemented by a local computing device (e.g., a smartphone, tablet, ordesktop computer) executing the user interface. Alternatively, Blocks ofthe second method S200 can be executed remotely, such as at a remoteserver interfacing with a local computing device to serve images to auser and to receive image filter parameters, image selections, and otherinputs from the user.

2.2 Images

Block S210 of the second method S200 recites displaying a first image ofa first assembly unit within a user interface, wherein a form of thefirst image was recorded at an optical inspection station. Generally, inBlock S210, the system retrieves a first image of a first assembly unitand presents this first image to the user through the user interface; atthe user interface, the user may then zoom in to various regions of thefirst image and shift the first image vertically and horizontally withina zoom window to visually, remotely inspect regions of the firstassembly unit represented in these regions of the first image.

2.2.1 Homography Transform

In one implementation, in Block S210, the system retrieves a firstdigital photographic image—previously recorded by an optical inspectionstation during an assembly period—from a database. The system thennormalizes the first digital photographic image to generate the firstimage that can then be presented to the user at the user interface. Forexample, the optical inspection station can include a digitalphotographic camera and a wide-angle lens coupled to the digitalphotographic camera; images recorded by the optical inspection stationmay therefore exhibit perspective distortion. During setup of theoptical inspection station, a reference object defining a referencesurface—such as a 300-millimeter-square white planar surface with blackorthogonal grid lines at a known offset distance of 10 millimeters—canbe placed within the optical inspection station, and the opticalinspection station can record a “reference image” of the referencesurface and upload the reference image to the remote database. Thesystem can then: retrieve the reference image; implement computer visiontechniques to identify warped grid lines in the reference image; andthen calculate an homography transform that maps warped grid lines inthe reference image to straight, orthogonal grid lines. In this example,the system can also calculate a scalar coefficient that relates adigital pixel to a real dimension (i.e., a length value in real space)based on the known distances between grid lines on the referencesurface. Therefore, the system can apply the homography transform to thefirst digital photographic image to generate the “flattened” (or“dewarped”) first image and then display the first image—now withperspective distortion removed—within the user interface forpresentation to the user. As described below, the system can alsoextract a real dimension of a feature of the first assembly unit fromthe first image by summing a number of pixels in the first imagespanning the feature and then multiplying this number of pixel by thescalar coefficient.

In the foregoing implementation, the system can transform all otherdigital photographic images recorded at the same optical inspectionstation during an assembly period for a particular assembly type at aparticular assembly stage according to the same homography transform;the system can also apply the same scalar coefficient to the resultingflattened images. For example, upon receipt of a new digitalphotographic image from the optical inspection station, the system can:immediately calculate a corresponding flattened image based on thehomography transform that is specific to this optical inspectionstation; and then store the original digital photographic image and thecorresponding flattened image together in the database. As describedbelow, the system can also generate a measurement space for the originaldigital photographic image, a compressed form (e.g., a thumbnail) of theflattened image, a feature space for the flattened image, and/or otherimages, spaces, or layers related to the digital photographic image orthe flattened image and store these data together (e.g., in one fileassociated with the corresponding assembly unit) in the database.Alternatively, the system can store the digital photographic image inthe database and then generate the corresponding flattened image in realtime when review of the corresponding assembly unit is requested at theuser interface.

2.2.2 Set of Images

The system can define a set of related images by assembly unitsrepresented in these images. For example, an optical inspection stationcan store a timestamp and an optical inspection station identifier inmetadata of an image; the system can also write an assembly type and anassembly stage to the image metadata based on the known location of theoptical inspection station along an assembly line. The system can alsoimplement computer vision techniques to read a serial number or otheroptical identifier from a region of the image representing an assemblyunit or a fixture locating the assembly unit with the optical inspectionstation and can write this serial number of other identifier to theimage metadata. Furthermore, the system can determine a configuration ofthe assembly unit represented in the image based on the timestamp,serial number, and/or assembly stage, etc. of the assembly unit andwrite this configuration to the image metadata. Similarly, the systemcan implement computer vision techniques (e.g., template matching,pattern matching, object recognition) to extract an assembly type and/oran assembly state of the assembly unit represented in the image directlyfrom the image. The system can repeat this process asynchronously forimages stored in the database and in (near) real-time for new imagesreceived from deployed optical inspection stations.

The system can then apply various filters to metadata stored with theseimages to define a set of related images. For example, the system canautomatically aggregate all images recorded at one optical inspectionstation during one assembly period or “build” (e.g., EVT, DVT, or PVT)to define the set of images. Similarly, the system can select—from abody of images recorded across a set of optical inspection stations andrepresenting multiple assembly units in various assembly states—a set ofimages representing a set of assembly units of the same assembly typeand in the same assembly state. The system can also receive a set offilter parameters from the user—such as time window, configuration,build, assembly stage, and/or other filters, as described below—andpopulate the set of images according to the filter. Furthermore, thesystem can order images in the set, such as by timestamp or serialnumber, and display these images in this order as the user scrollsthrough the set of images within the user interface.

2.3 View Window

Block S212 of the second method S200 recites, in response to a change ina view area of the first image at the user interface, displaying a firstsubregion of the first image within the user interface. Generally, inBlock S212, the system receives an input at the user interface,interprets this input as a command to change the view window of thefirst image currently rendered within the user interface, and updatesthe view window accordingly.

In one implementation, the system initially displays the full height andwidth of the first image within the user interface in Block S210. Uponreceipt of a zoom input—such as via a scroll wheel, selection of a zoomlevel from a dropdown menu, or a zoom level slider—at the userinterface, the system redefines a view window to encompass a smallerregion of the first image and renders the smaller region of the firstimage bound by this view window at higher resolution within the userinterface in Block S212. The system can then implement methods andtechniques described below to select a virtual origin within this newview window and to define geometry and location parameters of the viewwindow relative to the virtual origin.

Once zoomed into the first image, the user may drag or shift the firstimage vertically or horizontally relative to the view window. The systemcan then select a new virtual origin within the revised view windowand/or redefine geometry and location parameters of the revised viewwindow relative to the current virtual origin. Following a change inzoom level and each change in position of the first image relative tothe view window, the system can implement this process automatically: toupdate a region of the first image and a resolution of this regiondisplayed in the user interface; to reselect a virtual original of thefirst image (e.g., if the previous virtual origin is not longer withinthe view window); and to automatically recalculate geometry and locationparameters of the view window relative to the current virtual origin.

Alternatively, the system can: automatically update a region of thefirst image and a resolution of this region displayed in the userinterface in real-time in response to a change in zoom level andposition of the first image within the view window; and select a virtualoriginal of the first image and recalculate geometry and locationparameters of the view window relative to the virtual origin selectivelyin response to manual entry—through the user interface—of a command tostore the current view window and to populate the view window acrossother images in the set.

However, the system can implement any other methods or techniques toupdate a region and resolution of the first image rendered within theuser interface and to automatically or selectively trigger selection ofa virtual origin in the first image in Block S220 and recordation ofview window parameters in Block S230.

2.4 First Image: Origin Selection

Block S220 of the second method S200 recites locating a first virtualorigin at a first feature on the first assembly unit represented in thefirst image in Block S220. Generally, in Block S220, the system locatesa virtual origin within the first image relative to (e.g., on) adistinguishable feature within the first image; the system can thendefine the view window for the current region of the first imagerendered in the user interface relative to this virtual origin in BlockS230. Because the system locates the virtual origin at a distinguishablefeature within the first image, the system can implement computer visiontechniques to identify analogous (e.g., similar) distinguishablefeatures in other images in the set and to similarly locate virtualorigins relative to (e.g., on) these analogous features. By renderingimages in the set with their virtual origins at the same position withinthe user interface and at the same scale and resolution as the firstimage, the system can preserve a view window set for the first imageacross other images in the set, thereby enabling the user to view (e.g.,scroll through) regions of images of different assembly units positionedrelative to a common feature represented in these images.

In particular, locations of like parts, components, and subassembliesmay not be positioned in identical locations and orientations—relativeto the global assembly unit and to other parts, components, andsubassemblies within the assembly unit—across a group of assembly unitsassembled along the same assembly line over time. Furthermore, a fixtureconfigured to constrain an assembly unit within an optical inspectionstation may exhibit a non-zero location tolerance such that assemblyunits captured in a sequence of images may be noticeably offset from oneimage to the next. To preserve a view window from a first image of afirst assembly unit to a second image of a second assembly unit, thesystem can define a first virtual coordinate system within the firstimage, such as including a virtual origin and a virtual axis, and definethe view window relative to this first virtual coordinate system. Thesystem can then define a similar second virtual coordinate system in thesecond image, such as relative to like features of the first and secondassembly units captured in these two images, and project the view windowonto the second image based on the second virtual coordinate system. Bydefining reference coordinate systems across the set of images relativeto or based on like features in assembly units represented in the set ofimages, the system can display these like features in the same positionwithin the user interface, thereby enabling the user to quickly,visually distinguish differences in the relative positions of otherfeatures in these assembly units relative to these like features as theuser indexes through this set of images within the user interface.

2.4.1 Manual Origin Selection

In one implementation, the system can select a global feature within thefirst image and define a global virtual origin within a full area of afirst image in Block S220. In one example, the system: renders the firstimage (e.g., the whole image or an expanded subregion of the firstimage) with the user interface; overlays curves on the first image inalignment with features of the first assembly unit shown in the firstimage; receives selection of one or more points, intersections of twocurves, entire curves, etc. from the user; and then transforms theuser's selection into a virtual origin and an axis of a virtualcoordinate system for the first image.

While viewing the first image within the user interface, the user caninitiate a new view window specification for propagation across the setof images, such as by selecting a tab or other input region within theuser interface. Prior to presentation of the first image to the user oronce the user initiates the new view window specification, the systemcan: implement edge detection and/or other computer vision techniques toidentify edges of real features on the first assembly unit representedin the first image; generate a feature space laid over the first image;and populate the feature space with colored (e.g., yellow, green)vectors aligned with and representing edges of corresponding realfeatures shown within the first image. In this implementation, thesystem can generate line or curve vectors representing edges of featuresof the first assembly unit shown in the first image; the system can alsointerpolate areas of features bound by three or more straight lines orone or more curves. Furthermore, the system can identify points withinthe first image such as ends of lines and curves, intersections of linesand curves, corners of areas, and/or centroids of areas (e.g., centersof square, rectangular, and circular areas). Once the user initiates thenew view window specification, the system can activate the featurespace, such as by rendering the feature space laid over the first imageor by highlighting vectors within the feature space near a cursor as theuser manipulates the cursor within the user interface. The system canthen store the user's selection of one or more vectors from the featurespace and transform these vectors into a virtual origin for the firstimage, as described below.

Similarly, while viewing the first image within the user interface (andafter initiating the new view window specification), the user can selecta pixel within the current view window. The system can then select apoint, line, or curve—from the feature space specific to the firstimage—nearest this pixel. Alternatively, upon receipt of selection of apixel, the system can implement methods and techniques described aboveto scan a region of the image around the selected pixel for edges,generate a set of vectors representing these nearby features, and thenselect a vector nearest the selected pixel. The system can also renderthis vector over the first image to indicate the particular feature.Alternatively, the system can prompt the user to select multiple (e.g.,three) pixels around a feature (e.g., a point, corner, or center of anarea) or along a feature (e.g., along an edge) within the first imageand then implement similar methods and techniques to identify a singlefeature (e.g., an edge, corner, or center of an area) nearest thesemultiple pixels. The system can then store this feature(s) selected bythe user and implement methods and techniques described below to definea virtual origin in the first image accordingly. The system cantherefore: identify a set of edge features on the first assembly unitwithin the first subregion of the first image; receive selection of apixel within the first subregion of the first image; select a firstfeature, from the set of edge features, nearest the pixel; and locatethe virtual origin on the first feature. However, the system caninterface with the user in any other way through the user interface toreceive manual selection of a reference feature with the first image orto receive selection of a reference origin within the first image.

In this implementation, if the user selects a point feature—such as anintersection of two curves (e.g., a corner), an end of a line, or acenter of a surface, either directly through the feature space orindirectly by selecting a pixel—within the first image, the system canset the virtual origin in the first image at this point feature. Thesystem can also define an axis for the first image. For example, for thepoint feature that falls at the end of one line or curve in the featurespace, the system can define a virtual axis intersecting the virtualorigin and tangent to this line or curve. Similarly, for the pointfeature that falls at the intersection of two curves in the featurespace (e.g., a corner), the system can define: a first virtual axisintersecting the virtual origin and tangent to the first curve; and asecond virtual axis intersecting the virtual origin and tangent to thesecond curve. Therefore, in this example, when locating the firstvirtual origin in the first image, the system can also locate the firstvirtual origin—of a first coordinate system—on the first feature withinthe first image, the system can also align a first axis of the firstcoordinate system to the first feature. However, for the point featurelocated within a surface, the system can identify a nearest line (e.g.,an edge) feature bounding the surface and align a virtual axis to thisline feature; alternatively, the system can detect an edge featuredefining a global bound (e.g., a bottom edge, a left side) of the firstassembly unit and align the virtual axis to this edge feature. Thevirtual axis and virtual origin can thus cooperate to define a virtualcoordinate system for the first image. However, the system can implementany other method or technique to place the virtual origin and virtualaxis in the first image.

2.4.2 Automated Global Origin Selection

Alternatively, the system can automatically detect a reference featureon the first assembly unit shown in the first image and then define thevirtual origin for the first image relative to this reference feature inBlock S220.

In one implementation, the system automatically calculates a defaultvirtual origin for images in the set. For example, the system can:implement machine vision techniques (e.g., edge detection, templatematching) to detect the maximal perimeter of the first unit shown in thefirst image; identify the upper left corner of the maximal perimeter ofthe first assembly unit; define a virtual origin for the first image atthis upper left corner; implement machine vision techniques to detect astraight edge nearest and/or intersecting the virtual origin; and thenalign a virtual axis of the virtual coordinate system to this neareststraight edge.

Similarly, the system can define a virtual origin at a feature on afixture—restraining the first assembly unit within the opticalinspection station—represented in the first image. For example, thesystem can: implement template matching or other object recognitiontechniques to detect a datum in region of the first image beyond theperiphery of the first assembly unit (e.g., the fixture), such as aquick-response code, engraved real coordinate system, a set of threepins or polished steel spheres, a corner of the fixture plate, or otherknown optical marker on the fixture plate; and then place the virtualorigin at this datum.

In another example, the system can: implement machine vision techniquesto identify an upper left corner of the perimeter of a topmostrectilinear component shown in the first image (i.e., a componentnearest the camera in the station at which the first image wascaptured); define the virtual origin of the first image at this upperleft corner; implement machine vision techniques to find a neareststraight edge on the topmost rectilinear component shown in the firstimage; and then align the virtual axis of the virtual coordinate systemof the first image to this straight edge, as shown in FIG. 3.

2.4.3 Automated Origin Selection within Zoom Window

As shown in FIG. 4, the system can implement similar methods andtechniques to automatically select a feature within the first imagebound by the view window and to define the virtual origin relative tothis feature, such as in response to a change in the view area of thefirst image at the user interface. In particular, in thisimplementation, the system can implement similar methods and techniquesto select a local feature within the region of the first image renderedwithin the user interface and defines a local virtual origin relative tothis local feature. For example, in response to each change in zoomlevel and viewing position of the first image during an image viewingsession, the system can recalculate a virtual origin and a virtual axis(e.g., an orientation of the virtual coordinate system) relative to afeature of the first assembly unit shown in the expanded region of thefirst image currently rendered within the user interface.

The system can implement methods and techniques described above to:detect edges within the first image generally or within the region ofthe first image currently bound by the view window; identify a set ofdiscrete surfaces on the first assembly unit bound by these edges;select a particular discrete surface exhibiting a greatest dimension,such as a greatest area or greatest length in the set of discretesurfaces; select a feature that bounds a portion of the particulardiscrete surface (e.g., a longest edge of the particular discretesurface represented in the region of the first image rendered in theuser interface); locate a virtual origin on this feature in the firstimage, such as at a topmost and/or leftmost end of this feature shownwithin the view window; and then align a virtual axis parallel ortangent to this feature. Alternatively, the system can: calculate acentroid of the particular surface; define the virtual origin of thefirst image at this centroid; detect an edge feature bounding theparticular surface, and align the virtual axis to this edge feature. Thesystem can therefore automatically place a virtual origin and a virtualaxis in the first image according to a largest surface in a region ofthe image bound by the current view window. The system can implementsimilar methods and techniques to detect a surface—represented withinthe first image—bound by the current view window and located nearest thecamera that captured the first image, as described above.

Similarly, the system can: detect a set of edge features within theregion of the first image bound by the current view window; detectintersections of these edge features (i.e., “corners”); locate a virtualorigin at an upper-leftmost corner detected in this region of the firstimage; and align a virtual axis to an edge feature intersecting thiscorner. Similarly, the system can locate a virtual origin at a cornernearest the center of the current view window.

The system can thus detect edge features defining bounds of parts withinthe first assembly unit, detect corners of these parts, and place avirtual origin at a corner of one of these parts represented within theregion of the image currently rendered within the user interface, suchas a largest part, a part nearest the upper-left corner of the viewwindow, or a part at the highest elevation within a region of the firstassembly unit represented in this region of the image.

Alternatively, the system can: implement computer vision techniques,such as object recognition or template matching, to identify differentparts or part types within a sector of the first assembly unitrepresented in the region of the first image bound by the current viewwindow; identify a common reference part or common reference part type;and then define the virtual origin for the first image relative to thecommon reference part or common reference part type. For example, thesystem can identify: an edge of a PCB; the radial center of a cameralens; a node of an antenna; and/or a fastener head or fastener borewithin the region of the first image. The system can then implementmethods and techniques described above and a predefined common referencepart or part type hierarchy to select a highest-order part or part type,to select a feature bounding or defining this part or part type, and todefine a virtual origin and virtual axis according to this feature. Forexample, the part type hierarchy can prioritize a fixture, then anassembly unit (e.g., an enclosure), a part (e.g., a PCB), a subpart(e.g., an integrated circuit or other chip mounted to a PCB), etc. Inthis example, the system can implement computing device techniques toidentify features representative of these part types within the regionof the first image bound by the view window and then locate the virtualcoordinate system on the highest-ranking part type identified in thisregion of the first image according to the part type hierarchy.

In the foregoing implementation, the system can develop and revise thepart or part type hierarchy over time. For example, the system canimplement machine learning techniques to: track and characterize manualfeature selections, as described above; detect patterns in these manualfeature selections; develop a model for detecting like features inimages; and refine the part or part type hierarchy to automaticallyselect representative features in regions of images of assembly unitsdisplayed within the assembly unit over time.

However, the system can implement any other method or technique toautomatically locate a virtual origin and/or a virtual axis within thefirst image relative to one or more features represented in the regionof the first image bound by the current view window in Block S220.Furthermore, the system can toggle between setting a global virtualorigin and setting a local virtual origin for a first image currentlyviewed by the user, such as based on a state of a virtual radio buttonrendered within the user interface or when the user zooms into the firstimage from a lowest zoom level.

2.5 View Window Specification: Last Viewed Area Parameters

Block S230 of the second method S200 recites recording a geometry and aposition of the first subregion of the first image relative to the firstvirtual origin. Generally, in Block S230, the system records parameterscharacterizing the current view window on the first image relative tothe virtual origin and/or virtual axis defined for the first image. Inparticular, the system can record a geometry and a position of thecurrent view window—defining a region of the first image currentlyrendered within the user interface—relative to the virtual origin of thefirst image. The system can then store these data in a new view windowspecification.

In one implementation, as the user zooms into and out of the first imageand repositions the first image vertically and horizontally in the userinterface, the system can store parameters defining the last area of thefirst image rendered in the user interface in Block S230. For example,the system can store: the pixel width and pixel height of a rectangularregion of a first image rendered in the user interface; the horizontaland vertical pixel offsets of the upper-left corner between thisrectangular region and the virtual origin of the first image; and anangular offset between one edge of the rectangular region and thevirtual axis of the first image in the new view window specification. Inanother example, the system can store the pixel coordinates of eachcorner of the rectangular region of the first image rendered in the userinterface relative to a virtual coordinate system defined by the virtualorigin and the virtual axis in the first image. The system can implementthese parameters to project the view window for the first image ontoother images in the set in Block S242.

The system can also write a zoom level at which the first image iscurrently being viewed and/or a ratio of real dimension to pixel sizefor the current zoom level. In Block S242, the system can implementthese data to set a zoom level for other images in the set or to scalethese images to match the zoom level or scale of the first image.

The system can also store a location of the feature(s) selected todefine the virtual origin and the virtual axis in the first image and/ora location of a narrow feature window containing this feature(s) withinthe first image, such as relative to the upper-leftmost corner of thefirst image, relative to a datum or other reference fiducial on thefixture shown in the first image, or relative to another global originof the first image. Similarly, the system can characterize thisfeature(s), such as by categorizing the feature as a corner, line,curve, arc, or surface and calculating a length, radius, or area of thefeature (e.g., in pixel-based units or real units). For example, thesystem can store these parameters in the new view window specification.The system can then implement these parameters in Block S240 to identifylike features in other images and to locate comparable virtual originsand virtual axes in these other images.

However, the system can store any other set of values representing thesize and position of a region of the first image last rendered in theuser interface.

Furthermore, the system can implement similar methods and techniques tolocate the view window of the first image relative to a feature or setof features within the first image directly in Block S230, such asrather than relative to an origin or coordinate system located on afeature.

2.6 Second Image: Origin Selection

Block S240 of the second method S200 recites locating a second virtualorigin relative to a second feature represented in a second image of asecond assembly unit, wherein the second feature is analogous to thefirst feature. (Block S240 can similarly recite, in response to receiptof a command to advance from the first image to a second image of asecond assembly unit at the user interface, locating a second virtualorigin at a second feature on the second assembly unit represented inthe second image, wherein the second feature on the second assembly unitis analogous to the first feature on the first assembly unit.)Generally, in Block S240, the system automatically identifies a secondfeature—in a second image of a second assembly unit—analogous (e.g.,similar in position and geometry) to a first feature selected in thefirst image to define a first virtual origin and/or first virtual axesin the first image. Once this second feature in the second image isidentified in Block S240, the system can define a second virtual originand second virtual axes for the second image based on this secondfeature. In particular, in Block S240, when the user scrolls from thefirst image to the second image, the system implements methods andtechniques described above to automatically identify a second referencefeature in the second image substantially identical to the firstreference feature selected in the first image and to automaticallydefine a virtual origin in the second image according to this secondreference feature.

In one example, the system executes Block S240 for all other images inthe set, the next five images and five preceding images in the set ofimages, or the next image and the preceding image in the set each timethe user adjusts the view window of the first image or saves a new viewwindow specification. Alternatively, the system can execute Block S240to identify an analogous feature in a second image once the useradvances (e.g., scrolls, tabs forward or backward) from the first imageto the second image.

2.6.1 Bounded Scan Area

In one implementation, the system: projects a boundary encompassing andoffset from the first virtual origin of the first image onto the secondimage; identifies a set of edge features on the second assembly unitrepresented within a region of the second image contained within thisboundary; identifies the second feature, in this set of edge features,exhibiting a second geometry approximately a first geometry of the firstfeature from the first image; and locates the second virtual origin onor relative to the second feature according to parameters implemented bythe system when locating the first origin in the first image. Forexample, to detect the second feature in the second image, the systemcan: detect a global origin in the second image; project the featurewindow (described above) stored in the new view window specificationonto the second window according to the global origin of the secondimage; and scan a region of the second image bounded by this featurewindow for a feature exhibiting a geometry similar to the geometry ofthe first feature of the first image.

Alternatively, the system can scan the second image in its entirety fora second feature analogous to the first feature selected in the firstimage.

In Block 240, the system can also detect and compare other referencefeatures in the first and second images (e.g., lower-right corners ofPCBs shown in the images), determine if the scale of the second imagediffers globally or locally from the first image based on relativepositions of reference features in the first and second image, and thenscale the second image as necessary to match a first view of the secondimage to a last view of the first image previously rendered in the userinterface.

Alternatively, the system can identify and locate multiple analogousfeatures across multiple images of multiple assembly units, such as infeature spaces unique to each image, prior to serving images in this setto the user interface for viewing, such as to speed a process ofprojecting a view area of one image onto another image—based onpositions of analogous features—as the user scrolls through images inthis set.

2.6.2 Analogous Feature Detection

In one implementation, the system: identifies a set of featuresrepresented in the second image (e.g., across the entire second image orwithin a region of the second image bounded by the feature window);selects a second feature, in the set of features, exhibiting dimensionalcharacteristics and geometric characteristics approximating dimensionalcharacteristics and geometric characteristics of the first feature; andthen locates a second virtual origin on the second feature within thesecond image in Block S240. For example, the system can implementpattern matching techniques to match the second feature in the secondimage to the first feature in the first image. Similarly, in theimplementation described above in which the system detects a first partwithin the first assembly unit and locates the first virtual origin onthis first part in Block S220, the system can: implement computer visiontechniques (e.g., template matching, object recognition) to identify aset of parts in the second assembly represented in the second image;select a second part, in the set of parts, that is analogous to thefirst part, such as representing a geometry and position relative toother parts in the second assembly unit that is similar to the geometryand position of the first part in the first assembly unit relative toother parts in the first assembly unit; and then locates the secondvirtual origin on the second features of the second part in the secondimage in Block S240, such as a corner of the second part similar to thecorner of the first part on which the first virtual origin was locatedin Block S220.

In the foregoing example, the system can: implement methods andtechniques to generate a second feature space for the second image;calculate a best alignment between the second feature space of thesecond image and the first feature space of the first image; and selectthe second feature—from the second feature space—that falls near thefirst feature in the aligned first feature space and exhibits a geometry(e.g., length and form) similar to that of the first feature.

However, the system can implement any other methods and techniques toautomatically detect a second feature (e.g., a point, line, curve, orarea feature) in the second image that is analogous to the first featurein the first image (i.e., the first feature by which the system locatedthe first virtual origin and first virtual axis in the first image). Thesystem can then replicate the process executed in Block S220 to locatethe first virtual origin and the first virtual axis relative to thefirst feature in the first image to locate a second virtual origin and asecond virtual axis (e.g., a second coordinate system) relative to thesecond feature in the second image in Block S240.

2.7 View Window Projection and Display

Block S242 recites projecting the geometry and the position of the firstsubregion of the first image onto the second image according to thesecond virtual origin to define a second subregion of the second image;and Block S250 recites displaying the second subregion of the secondimage within the user interface. Generally, in Block S242, the systemselects a region of the second image to initially render within the userinterface based on a set of parameters defining the last region of thefirst image previously rendered within the user interface. Inparticular, in Block S242, the system projects the last view window ofthe first image onto the second image based on the second virtual originof the second image to define an analogous view window for the secondimage. Therefore, when the system replaces the region of the first imagebound by the view window with the second image bound by the analogousview window within the user interface in Block S250, the second featurein the second image—which is analogous to the first feature in the firstimage—is rendered at the same location and in the same orientationwithin the user interface as was the first feature immediately prior.Therefore, the system can locate coordinate systems in the first andsecond images in Blocks S220 and S240, respectively, based on analogousreference features common to the first and second image and project aview window from the first image onto the second image in Block S242such that the reference feature in the second image is aligned intranslation and rotation with its analogous reference feature in thefirst image when the subregion of the second image is rendered withinthe user interface in replacement of the subregion of the first image inBlock S250. For example, the system can render the subregion of thesecond image in the user interface in Block S250 substantially inreal-time as the user scrolls from the first image to the second image.The system can later repeat Blocks S240, S242, and S250 as the userscrolls from the second image back to the first image or from the secondimage to a third image—of a third assembly unit—in the set of images.

In one implementation, the system: defines a geometry of a view windowfor the second image according to the zoom level stored in the new viewwindow specification; vertically offsets an origin of the view windowfrom the second virtual origin in the second image according to avertical offset stored in the new view window specification;horizontally offsets the origin of the view window from the secondvirtual origin in the second image according to an horizontal offsetstored in the new view window specification; rotates the view windowrelative to the second coordinate system in the second image accordingto an angular offset stored in the new view window specification; anddefines a region of the second image bound by the new view window as thesecond subregion in Block S242. The system can therefore both translateand rotate the view window in the second image to align with the viewwindow in the first image to compensate both for variations in localpositions and orientations of parts within the first and second assemblyunits and global variations in the positions and orientations of thefirst and second assembly units within the optical inspection stationwhen the first and second images were recorded. The system can thendisplay the second subregion of the second image within the userinterface in replacement of the first subregion of the first image inBlock S250.

Alternatively, the system can implement similar methods and techniquesto locate the view window from the first image onto a second imagerelative to a second feature or set of features within the second imagein Block S242. However, the system can implement any other method ortechnique to project the view window of the first image onto the secondimage in Block S242 and to display the corresponding region of thesecond image within the user interface with the reference feature of thesecond image aligned to the analogous reference feature of the firstimage previously displayed in the user interface.

2.8 View Window Propagation

In one variation, the system serves an indicator of the second feature,the second virtual origin, and/or the second virtual axis automaticallyselected for the second image in Blocks S240 and S242 and a prompt toconfirm these selections to the user through the user interface. Uponreceipt of confirmation of these selections from the user, the systemcan replicate the process implemented in Blocks S240 and S242 for allother images in the set.

In one implementation, once the system automatically selects a geometryand a position of the second subregion of the second image in Block S242and renders the second subregion of the second image within the userinterface in Block S250, the system serves a prompt to confirm thisgeometry and position of the second subregion to the user through theuser interface before executing these processes for other images in theset. If the user indicates through the user interface that this geometryand position are incorrect—such as angularly offset, shifted verticallyor horizontally, or improperly scaled relative to the first subregion ofthe first image displayed previously—the system can: repeat Blocks S240and S242 for the second image to recalculate the geometry and positionof the second subregion of the second image; display this revised secondsubregion of the second image in the user interface; and similarlyprompt the user to confirm the geometry and position of the revisedsecond subregion. Upon receipt of an indication that the secondsubregion of the second image is improper, the system can also promptthe user to select an alternate second feature in the second imageand/or indicate a preferred origin, axis, and/or coordinate system forthe second image, such as by selecting an alternate pixel within thesecond image or selecting an alternate feature from a second featurespace laid over the second image as described above. The system can thenrevise the second subregion of the second image and update a process forpropagating the view window across the set of images, such as stored inthe new view window specification, according to these additionalselections entered by the user. For example, the system can implementmachine learning techniques to refine a process or model forautomatically selecting analogous features, locating virtual origins,and orienting virtual axes across a set of related images based onfeedback provided by the user.

However, in response to receipt of confirmation of the projectedgeometry and the projected position of the second subregion of thesecond image, the system can: retrieve the set of images of otherassembly units from the database as in Block S210; locate a virtualorigin in each image in the set of images as in Block S240; and projectthe geometry and the position of the first subregion of the first imageonto each image in the set of images to define a set of subregions forthe set of images in Block S242. In particular, once the user confirmsthat the system correctly defined the second subregion in the secondimage, the system can propagate the last view window of the first image,such as defined in the new view window specification, across all imagesin the set. The system can then index through the set of subregionsdisplayed within the user interface in response to a scroll input at theuser interface as in Block S250 described above. However, the system canimplement any other methods or techniques to prompt, collect, andrespond to user feedback related to automatic selection of the secondsubregion of the second image.

As described above, the system can execute Blocks S240 and S242 for allremaining images in the set once the user confirms the second subregionof the second image, such as before the user scrolls to or selects anext image in the set. Alternatively, the system can execute theforegoing methods and techniques to propagate the last view window ofthe first image to images in the set in real-time as the user indexesforward and backward to these other images within the user interface.

2.9 Composite Image

One variation of the second method S200 includes Block S252, whichincludes virtually aligning images in the set by their coordinatesystems, reducing opacities of these images, and overlaying these imagesto form a composite image. Generally, in this variation, the system can:virtually stack two (or more images)—from the set of images—with theiranalogous features or analogous-feature-based coordinate systems inalignment; reduce the opacity of these images to form a composite image;and display this composite image within the user interface. Thus, whenviewing the composite image, the user may view deviations in positionsand geometries of like components (e.g., housings, sub-assemblies,parts, subparts) of assembly units represented in these images relatedto a common reference feature.

For example, in Block S252, the system can: set a first opacity of thefirst subregion of the first image; set a second opacity of the secondsubregion of the second image; overlay the second subregion over thefirst subregion to generate a composite image; and display the compositeimage within the user interface. The system can apply a static opacity,such as 50% opacity, to each image when generating the composite image.Alternatively, the system can enable the user to dynamically adjust theopacity of images represented in the composite image and then update thecomposite image rendered in the display accordingly. For example, thesystem can: present a slider bar adjacent the composite image displayedin the user interface; adjust the first opacity of the first imageaccording to a change in the position of a slider on the slider bar;adjust the second opacity of the second image as an inverse function ofthe first opacity; and refresh the composite image accordingly.

The system can implement similar methods and techniques to align andcombine two or more whole images from the set of images into a compositeimage.

In another implementation, the system generates a composite image froman image of a real assembly unit and an image of a graphical modelrepresenting an assembly unit. In this implementation, by aligning animage of a real assembly unit to an image of the graphical model thatrepresents the assembly unit within a single composite image and thenrendering this composite image within the user interface, the system canenable the user to quickly visually distinguish differences in componentpositions and orientations between the real assembly unit and a nominalrepresentation of the assembly unit defined in the graphical model. Forexample, the system can: retrieve a virtual three-dimensionalcomputer-aided drafting (“CAD”) model representing the first assemblyunit; generate a two-dimensional CAD image of the CAD model at anorientation and in a perspective approximating the orientation andposition of the first assembly unit represented in the first image;locate a third virtual origin at a third feature—analogous to the firstfeature on the first assembly unit—in the CAD image, such as byimplementing methods and techniques similar to Block S240 describedabove; projecting the geometry and the position of the first subregionof the first image onto the virtual CAD model according to the thirdvirtual origin to define a third image, such as by implementing methodsand techniques similar to Block S242 described above; and then display atranslucent form of the third image over the first subregion of thefirst image within the user interface. Thus, in this example, the systemcan align the CAD image to the first image in rotation and translationby a real feature on the real assembly unit represented in the firstimage and a graphical feature representing the real feature in the CADmodel.

Alternatively, the system can implement similar methods and techniquesto: generate a CAD image; project the view window from the first imageonto the CAD image to define a subregion of the CAD image analogous tothe first subregion of the first image; and to display the subregion ofthe CAD image within the user interface independently of the firstimage, such as when the user scrolls from the first image to the CADimage while the new view window specification is active.

2.10 One Assembly Unit at Different Assembly Stages

In one variation, the system implements similar methods and techniquesto preserve a viewing window across a set of images of a single assemblyunit throughout a sequence of assembly stages. For example, in BlockS120, the system can assign a virtual origin to a first image based on afeature corresponding to a largest physical body shown in the firstimage (e.g., a corner of a PCB, a corner or perpendicular sides of arectangular housing). In this example, the system can identify the samefeature in other images of the assembly unit at various assembly stagesand assign like virtual origins to these other images.

In this variation, the second method S200 can include: displaying afirst image of an assembly unit in a first stage of assembly within auser interface in Block S210, the first image recorded at a firstoptical inspection station; locating a first virtual origin in the firstimage at a feature on the assembly unit represented in the first imagein Block S220; in response to receipt of a zoom input at the userinterface, displaying a first subregion of the first image within theuser interface in Block S212; storing a geometry and a position of thefirst subregion of the first image relative to the first virtual originin Block S230; identifying the feature on the assembly unit in a secondimage of the assembly unit in a second stage of assembly in Block S240;locating a second virtual origin in the second image according to thefeature in Block S240; defining a second subregion of the second imagebased on the geometry and the position of the subregion of the firstimage and the second virtual origin in Block S242; and, in response toreceipt of a command to advance from the first image to the second imageat the user interface, displaying the second subregion of the secondimage within the user interface in Block S250.

For example, the system can: retrieve a first digital photographicimage—recorded by a first optical inspection station at a first positionalong an assembly line—of the first assembly unit from a database;normalize the first digital photographic image to form the first image,as described above; retrieve a second digital photographicimage—recorded by a second optical inspection station at a secondposition along the assembly—of the first assembly unit; and normalizethe second digital photographic image to form the second image. Thesystem can then implement methods and techniques described above todefine analogous subregions of the first and second images (and otherimages of the first assembly unit) and to sequentially display thesesubregions as the user indexes through these images.

In particular, in this variation, the system can implement methods andtechniques described above to display expanded views of the samephysical position of a single assembly unit from a sequence of imagesrecorded at various stages of the assembly unit's assembly. By aligningimages of one assembly unit at different stages of assembly by a commonfeature and sequentially displaying these images in the user interfaceresponsive to scroll or index inputs entered by the user, the system canenable the user to view changes to the assembly unit over time (e.g.,along the assembly line) with a view window of these separate imageslocked to a common reference feature contained in these images.

2.10.1 Obscured Alternate Reference Features

In this variation, if a reference feature selected in a first image todefine a virtual origin in the first image is not visually available ina second image of the same unit, such as due to obscuration by acomponent installed on the assembly unit between capture of the firstimage and capture of the second image, the system can: select analternate feature on the assembly unit that is visually available inboth the first and second images when as the user scrolls from the firstunit to the second unit; and then redefine virtual origins (or assignsecondary or backup virtual origins) for the first and second images ofthe assembly unit.

In particular, the system can: align a first image (or a subregion ofthe first image) of the assembly unit to a second image (or a subregionof the second image) by one common feature represented in both the firstand second images; locate a third virtual origin in the second image ata second feature on the assembly unit represented in the second image;store a geometry and a position of the second subregion of the secondimage relative to the third virtual origin; identify the second featureon the assembly unit in a third image; locate a fourth virtual origin inthe third image according to the second feature; define a thirdsubregion of the third image based on the geometry and the position ofthe second subregion of the second image and the fourth virtual origin;and display the first, second, and third subregions of the first,second, and third images as the user advances through the assemblystages of the assembly unit at the user interface. For example, thesystem can align a first subregion of a first image to a secondsubregion of a second image by a corner of a PCB within the assemblyunit shown in both the first and second subregions of the first andsecond image. In this example, an enclosure is installed on the assemblyunit prior to recordation of a third image, thereby obscuring the PCB.Thus, to align the third image to the second image, the system candetect an edge of an housing of the assembly unit shown in both thesecond and third images and align the second and third images accordingto the edge of the housing.

2.10.2 Transparent Composite View

In this variation, the system can implement methods and techniquesdescribed above to compile two or more images (or subregions of two ormore images) of one assembly unit at different assembly stages into asingle composite image. For example, the system can: set a first opacityof a first subregion of a first image of the assembly unit at a firstassembly stage; set a second opacity of a second subregion of a secondimage of the assembly unit at a second (e.g., later) assembly stage;align the first and second subregions of the first and second images bya common feature of the assembly unit represented in both the first andsecond subregions; merge the first and second subregions into acomposite image; and then display the composite image within the userinterface. In this example, the system can compile multiple (e.g., all)images of one assembly unit into a static composite image and renderthis within the user interface. Alternatively, the system can compilethese images into a dynamic composite image. For example, the system caninitially display the composite image with a first image of the assemblyunit in a first assembly state shown at 100% opacity and all otherimages at 0% opacity; as the user scrolls through the composite image,the system can decrease the opacity of the first image and increase theopacity of a second image of the assembly unit in a second assemblystate; once the second image is shown at 100% opacity and as the usercontinues to scroll through the composite image, the system can decreasethe opacity of the second image and increase the opacity of a thirdimage of the assembly unit in a third assembly state; etc. until a lastimage a set of images of the assembly unit is shown at full opacity.

3. Optical Measurement

As shown in FIG. 6, a third method S300 for automatically generating acommon measurement across multiple assembly units includes: detecting afirst set of features in a first assembly unit shown in a first imagecaptured by an optical inspection station in Block S310; at a userinterface, displaying the first image and a set of curves over the setof features in the first image in Block S320; and generating ameasurement specification for a particular feature in assembly unitsimaged by the optical inspection station based on a curve manuallyselected from the set of curves within the user interface in Block S330.The third method S300 also includes, for each image in a set of imagescaptured by the optical inspection station (the set of images includingthe first image): identifying the particular feature in an assembly unitshown within the image based on the measurement specification in BlockS340; mapping a distorted measurement space onto the image in BlockS344; and calculating a real dimension of the particular feature basedon the geometry of the feature within the distorted measurement space inBlock S344. Finally, the third method S300 can include generating agraphical plot of real dimensions of the particular feature in assemblyunits imaged by the optical inspection station in Block S350, as shownin FIGS. 7 and 8.

One variation of the third method S300 for automatically generating acommon measurement across multiple assembly units includes: displaying afirst image within a user interface in Block S320, a form of the firstimage recorded at an optical inspection station; receiving manualselection of a particular feature in a first assembly unit representedin the first image in Block S310; receiving selection of a measurementtype for the particular feature in Block S330; and extracting a firstreal dimension of the particular feature in the first assembly unit fromthe first image according to the measurement type in Block S344. Thethird method S300 also includes, for each image in a set of images:identifying a feature in an assembly unit represented in the image inBlock S340, the feature in the assembly unit analogous to the particularfeature in the first assembly unit; and extracting a real dimension ofthe feature in the assembly unit from the image according to themeasurement type in Block S344. The third method S300 further includesaggregating the first real dimension and a set of real dimensionsextracted from the set of images into a digital container in Block S350.

Another variation of the third method S300 includes retrieving a set ofimages in Block S310 and, for a first image in the set of images:displaying the first image within a user interface, a form of the firstimage recorded at an optical inspection station, and displaying thefirst real dimension with the first image within the user interface inBlock S320; receiving manual selection of a particular feature in afirst assembly unit represented in the first image in Block S310;determining a measurement type for the particular feature in Block S330;and extracting a first real dimension related to the particular featurein the first assembly unit from the first image according to themeasurement type. The third method S300 also includes, for a secondimage in the set of images: automatically identifying a second featurein a second assembly unit represented in the second image in Block S340,the second feature in the second assembly unit analogous to theparticular feature in the first assembly unit; and extracting a secondreal dimension related to the second feature in the second assembly unitfrom the second image according to the measurement type in Block S344.In this variation, the third method S300 further includes, for a thirdimage in the set of images: automatically identifying a third feature ina third assembly unit represented in the third image in Block S340, thethird feature in the third assembly unit analogous to the particularfeature in the first assembly unit; and extracting a third realdimension related to the third feature in the third assembly unit fromthe third image according to the measurement type in Block S340. Thethird method S300 also includes: in response to selection of the secondimage at the user interface, displaying the second image and the secondreal dimension within the user interface in Block S320; and, in responseto selection of the third image at the user interface, displaying thethird image and the third real dimension within the user interface inBlock S320.

3.1 Applications

Generally, the system can execute Blocks of the third method S300: toreceive, from a first image of a first unit, a selection of a realfeature in the first unit shown; to define a call to calculate a realdimension (e.g., length, radius, parallelism, etc.) of the feature fromthe first image; to propagate the call across other images of similarunits; and to automatically calculate real dimensions of like featuresacross multiple like units from corresponding images of the assemblyunits. The system can then assemble these real dimensions of one featuretype from multiple units into a graphical plot, histogram, table, trendline, or other graphical or numeric representation of real dimensions ofthe feature across the group of units. The system also calculates a realdimension of a feature in a unit by implementing machine visiontechniques (e.g., edge detection) to identify the feature in an image ofthe assembly unit, mapping a two- or three-dimensional measurement spaceto the image in order to compensate for optical distortion in the image,and then calculates a real dimension of the feature based on itsposition in the image relative to the measurement space.

The system can therefore execute Blocks of the third method S300 toretroactively and automatically configure measurement of a batch ofunits from images of the assembly units after the images are captured,thereby eliminating a need to configure optical inspection stations onan assembly line before assembly of units on the line. The system canalso execute Blocks of the third method S300 to enable one or more usersto create and access new measurements of real unit features from oldimages of units without necessitating that new images of the assemblyunits be captured and without necessitating manual selection of the samefeature for measurement across images of multiple like units. The systemcan similarly execute Blocks of the third method S300 in real-time, suchas locally at an optical inspection station, in a remote server, or on auser's computing device (e.g., smartphone).

Blocks of the third method S300 can be executed locally by a computingdevice, such as a smartphone, tablet, or desktop computer. The systemcan include a display and render a user interface on the display toreceive selection of a feature for measurement in an image and topresent real dimension results to the user. Blocks of the third methodS300 can additionally or alternatively be executed by a remote computersystem, such as a remote server, that interfaces with a local computingdevice (e.g., a smartphone, tablet, or desktop computer) to receiveselection of a feature for measurement in an image and to present realdimension results to the user. However, any other local or remotecomputing device, computer system, or computer network (hereinafter“system”) can execute Blocks of the third method S300.

The third method S300 is described herein as implemented by the systemto propagate a measurement specification for a dimensional value (e.g.,length, radius, parallelism, etc. in metric or Imperial units). However,the third method S300 can additionally or alternatively be implementedto define a part-present specification, to propagate the part-presentspecification across images of multiple units, and to confirm that acomponent specified in the part-present specification is or is notpresent in each image. Similarly, the third method S300 can beimplemented to define a text specification, a color specification, acategory of a marking (e.g. production lot code) specification, or aspecification of any other type, to propagate these specificationsacross images of multiple units, and to confirm that a text string, acolor, a marking, or other feature specified in the part-presentspecification is or is not present in each image.

3.2 Images

Block S320 of the third method S300 recites displaying a first imagewithin a user interface. Generally, in Block S320, the system canimplement methods and techniques described above in Block S210 toretrieve a first image from a database and to present the first image toa user through the user interface.

As described above in Block S210, the system can also: normalize (or“flatten,” “dewarp”) the first image and other images stored in adatabase (or “body”) of images; aggregate a set of related images, suchas images of various assembly units of the same type and in the sameassembly state or a set of images representing the first (i.e., asingle) assembly unit in various stages of assembly.

3.3 Feature Selection

Block S310 of the third method S300 recites receiving manual selectionof a particular feature in a first assembly unit represented in thefirst image. Generally, in Block S310, the system interfaces with theuser through the user interface to receive a selection of a particularfeature or set of features from which the system subsequently extracts adimension in Block S344.

3.3.1 Feature Space and Vector-Based Selection

In one implementation, the system: implements computer vision techniquesto identify features of the first assembly unit represented in the firstimage; generates a feature space containing vectorized points, lines,curves, areas, and/or planes representing these features; and overlaysthe first image with this feature space, as described above. Inparticular, when an image of a unit captured by an optical inspectionstation is selected by a user for insertion of a measurement, the systemcan implement machine vision techniques to automatically detect featuresof the assembly unit shown in the first image in Block S310. Forexample, the system can implement edge detection techniques to identifycorners (e.g., points), edges (e.g., lines, curves), and surfaces (e.g.,areas, planes) in the first image. In Block S320, to guide the user inselecting one or more features in the first image for measurement, thesystem can: generate the feature space—specific to the first image—thatcontains vectorized points, curves, areas, and/or planes aligned topoints, lines, and surfaces detected in the first image; and then renderthe first image and the feature space laid over the first image in theuser interface, as shown in FIG. 6. The system can then receive manualselection of a particular vector (or a set of vectors) from the firstset of vectors contained in the first feature space from the user viathe user interface and then identify the particular feature (or set offeatures) corresponding to the particular vector(s) selected by theuser.

3.3.2 Pixel-Based Feature Selection

Alternatively, the system can interface with the user interface toreceive selection of a pixel from the first image and to implementmethods and techniques described above to select a particularfeature—from a set of features—in the first image nearest or otherwisecorresponding to this pixel. For example: while viewing the first imagewithin the user interface, the user can navigate a cursor to a pixelnear a desired corner feature, near a desired edge feature, or on adesired surface and select this pixel; as described above, the systemcan then compare this pixel selection to a feature space specific to thefirst image to identify a particular feature nearest the selected pixel.

As described above, the system can also prompt the user to selectmultiple pixels nearest a desired corner, along a desired edge, or on adesired surface represented in the first image; the system can thencompare these selected pixels to the feature space to select a corner,line (or curve), or area that best fits the set of selected pixels.However, the system can interface with the user through the userinterface in any other way to receive a selection of a particularfeature from the first image in Block S310. The system can implementsimilar methods and techniques to receive selections of multipledistinct features from the first image, as described below.

The system can implement subsequent Blocks of the third method S300 todefine a measure specification for the set of images based on thisfeature(s), to extract a real dimension of this feature from the images,and to populate this measurement specification across other images inthe set.

3.4 Measure Specification

Block S330 of the third method S300 recites receiving selection of ameasurement type for the particular feature. Generally, in Block S330,the system generates a measurement specification defining a measurementtype for the feature(s) selected in Block S310 and characterizing theparticular feature(s) selected from the first image.

As described above, the system can receive selection of one or morevectorized curves in the feature space from the user. For example, fromvectorized curves contained in the feature space overlaid on the firstimage, the user can select a vectorized point, an intersection of twovectorized curves, a single vectorized curve, two non-intersectingvectorized curves, or an area enclosed by one or more vectorized curves.The system can populate a measurement type menu within the userinterface with various measurement types, such as distance (e.g.,corner-to-corner), length (e.g., end-to-end or edge length), radius (ordiameter), planarity, parallelism, circularity, straightness, lineprofile, surface profile, perpendicularity, angle, symmetry,concentricity, and/or any other measurement type for the particularfeature(s) in the first image; the user can then select a measurementtype for the selected points, intersections, curves, and/or areas in thefeature space from this menu.

Based on the type(s) of features selected by the user, the system canalso filter, order, and/or suggest a measurement type in a set ofsupported measurement types. For example, upon selection of a singleline (e.g., a substantially straight curve) in Block S310, the systemcan predict a length-type measurement and can enable a length-typemeasurement type in the menu of measurement types accordingly. Uponselection of an arc in Block S310, the system can enable a total arclength measurement, a radius measurement, and a diameter measurement inthe menu of measurement types. Upon selection of an area in Block S310,the system can enable a total area measurement and a circumferencemeasurement in the menu of measurement types. Upon selection of a pointand a curve in Block S310, the system can enable a nearest distancemeasurement and an orthogonal distance measurement in the menu ofmeasurement types. Upon selection of a first curve and a second curve inBlock S310, the system can enable a nearest distance measurement, anangle measurement, and a gap profile measurement (e.g., a gap distanceas a function of length along the first and second curves) in the menuof measurement types. Upon selection of three points in Block S310, thesystem can enable a measurement for calculating a smallest circle formedby the three points in the menu of measurement types. However, thesystem can support any other predefined or user-defined (e.g., custom)measurement type. The system can also receive a selection for ameasurement type or automatically predict a measurement type for theparticular feature(s) in any other way.

From the particular feature(s) (e.g., an original feature in the firstimage or a vectorized point, curve, and/or area, etc. in the featurespace specific to the first image) selected from the first image, thesystem can generate a measurement specification for the set of images inBlock S330. For example, in Block S330, the system can define a featurewindow containing the particular feature in the first image and storethis position and geometry of the feature window (e.g., relative to anorigin of the first image or relative to the upper-left corner of thefirst image) in the measurement specification; as shown in FIG. 6, thesystem can project this feature window onto other images in the set inBlock S340 to identify features—analogous to the particular featureselected from the first image—in these other images in the set. Inparticular, when processing the set of images according to themeasurement specification in Block S340, the system can project thefeature window onto each image in the set and can then scan regions ofthese images bound by the feature window for features analogous to thefeature(s) selected in Block S310. In this implementation, the systemcan implement methods and techniques described above in the secondmethod S200 to align the feature window defined at the first image toother images in the set.

The system can also characterize the particular feature and store thischaracterization in the measurement specification. For example, thesystem can: implement template matching or pattern recognitiontechniques to characterize the particular feature as one of an arc,spline, circle, or straight line; write this characterization of theparticular feature to the measurement specification in Block S330; andapply this characterization to other images in the set to identifyfeatures of the same type in these other images in Block S340. Thesystem can also: calculate a real or pixel-based dimension of theparticular feature; store this dimension in the measurementspecification; and detect analogous features—in the remaining images inthe set—that exhibit similar real or pixel based dimensions, such aswithin a tolerance of ±2% in Block S344. Similarly, the system can:prompt the user to enter a nominal dimension and a dimensional tolerancefor the nominal dimension for the particular feature(s), as shown inFIGS. 7 and 8; or extract the nominal dimension and dimensionaltolerance from a CAD model of the first assembly unit, as describedbelow; and identify features in other images in the set that areanalogous to the particular feature based on the nominal dimension ofthe feature.

The system can also prompt the user to enter a name of the measurement(e.g., “antenna_height_1”), a description of the measurement (e.g.,“antenna height”), tags or search terms for the measurement (e.g.,“John_Smiths_measurement_set,” “RF group,” “DVT” or “EVT”, etc.), and/orother textual or numerical data for the measurement. The system can thenstore these data in the measurement specification, as shown in FIGS. 6and 9C. For example, the system can enable the user to toggle such tagswithin the user interface to access and filter points represented in agraph or chart of real dimensions of analogous features read from imagesin the set according to the measurement specification. Similarly, thesystem can enable another user to search for this measurementspecification by entering one or more of these tags or other termsthrough a search window in another instance of the user interface inorder to access this measurement specification for this set of images orto access this measurement specification for application across anotherset of images. The system can therefore enable multiple users: to applygeneral, group-specific, and/or user-specific measurement specificationsacross various sets of images; to access data extracted from a set ofimages according to a general, group-specific, and/or user-specificmeasurement specification; and to access measurement specificationsconfigured by other users.

However, the system can collect any other related information from theuser or extract any other relevant data from the first image to generatea measurement specification for the set of images in Block S330. Thesystem can then apply the measurement specification across images in theset in Blocks S340 and S344 to identify analogous features in assemblyunits represented in these images and to extract real dimensions ofthese analogous features directly from these images.

3.5 Analogous Feature Detection

Block S340 of the third method S300 recites, for each image in a set ofimages, identifying a feature—analogous to the particular feature in thefirst assembly unit—in an assembly unit represented in the image. (BlockS340 can similarly recite identifying a second feature in a secondassembly unit represented in the second image, the second feature in thesecond assembly unit analogous to the particular feature in the firstassembly unit.) Generally, in Block S340, the system: scans an image inthe set of images for a feature analogous to the particular featureselected from the first image, such as a feature located in a similarposition and exhibiting a similar geometry (e.g., real or pixel-baseddimension, feature type) as the particular feature selected from thefirst image, such as by implementing methods and techniques describedabove in the second method S200; and repeats this process for remainingimages in the set to identify a corpus of analogous features in assemblyunits represented across the set of images. The system can then extractreal dimensions of these features directly from these images in BlockS344 and assemble these dimensions into a graph, chart, or statisticrepresenting dimensional variations of like features (e.g., lengths,width, radii of parts; relative positions of parts; gaps between parts;etc.) across this set of assembly units.

To calculate real dimensions of analogous features across a set ofimages of assembly units at the same or similar assembly stage in one ormore builds according to a single measurement specification configuredby the user, the system can scan each image in the set of images for afeature analogous (e.g., substantially similar, equivalent,corresponding) to the feature selected by the user and specified in themeasurement specification. For example, for each image selected forcalculation of a real dimension according to the measurementspecification, the system can implement methods and techniques describedabove to detect features in an image and then select a feature in theimage that best matches the relative size, geometry, position (e.g.,relative to other features represented in the image), color, surfacefinish, etc. of the particular feature selected from the first image andspecified in the measurement specification in Block S340. The system canthen calculate a dimension of the analogous feature for each image inthe set in Block S344, as described below.

3.5.1 Window Scan

In one implementation, the system: defines a feature window encompassingthe particular feature, offset from the particular feature, and locatedaccording to an origin of the first image; and stores the feature windowin the measurement specification, as described above in Block S330described above. For example, the system can define the feature windowrelative to a global origin of the image (e.g., an upper-left corner ofthe first image). Alternatively, the system can implement computervision techniques to detect the perimeter of the first assembly unit inthe first image, define an origin on the first assembly unit in thefirst image (e.g., at an upper-left corner of the first assembly unit),and define a location of the feature window relative to theassembly-unit-based origin. The system can also implement a presetoffset distance (e.g., 50 pixels) and define the geometry of the futurewindow that encompasses the particular feature and is offset from theparticular feature by the preset offset distance. For example, for theparticular feature that defines a corner, the system can define acircular feature window 100 pixels in diameter; for the particularfeature that defines a linear edge, the system can define a roundedrectangular feature window 100 pixels wide with corners exhibiting 50pixels in radius and offset from near ends of the linear edge by 50pixels.

The system can then locate the feature window within an image accordingto a global origin of the image (e.g., an upper-left corner of theimage). Alternatively, the system can repeat the process described aboveto define an assembly-unit-based origin within the image and to locatethe feature window in the image according to this assembly-unit-basedorigin. The system can then: scan a region of the image bound by thefeature window to identify a limited set of features in the image; andcompare geometries and sizes of features in this limited set of featuresto the characterization of the particular feature stored in themeasurement specification to identifying one feature in the image bestapproximating (e.g., “analogous to”) the particular feature from thefirst image. The system can repeat this process for each remaining imagein the set of images.

3.5.2 Feature Matching in Feature Spaces

In another implementation, the system generates a first feature spacefor the first image in Block S310, labels a particular vectorrepresenting the particular feature in the first feature space, andstores the first feature space in the measurement specification in BlockS330. The system can then implement similar methods and techniques toidentify a set of features in an image in the set of images and togenerate a feature space containing a set of vectors representing thisset of features in the image. The system can then: align the featurespace of the image to the first feature space of the first image;identify a vector in the set of vectors nearest the particular vector inthe first set of vectors in location and geometry; and label the featurein the image corresponding to the vector as analogous to the particularfeature in the first image. The system can repeat this process for eachremaining image in the set of images.

However, the system can implement any other method or technique toidentify like features—of assembly units represented in the set ofimages—that are analogous to the particular feature of the firstassembly unit selected from the first image. Furthermore, the system canrepeat the foregoing processes for each of multiple distance featuresselected from the first image in Block S310.

3.5.3 Feature Confirmation

In one variation, the system implements methods and techniques describedabove to receive confirmation from the user that its identification of asecond feature—in a second image in the set of images—is analogous tothe particular feature selected from the first image before repeatingthis process to identify analogous features in other images in the set.For example, once the measurement specification is defined in BlockS330, the system can: execute a feature selection routine to identify asecond feature in the second image predicted to be comparable (i.e.,analogous) to the particular feature selected from the first image; andstore steps of this feature selection routine or a characterization ofthe feature selection routine in memory (e.g., in the measurementspecification). Before repeating the feature selection routine for otherimages in the set, the system can: display the second image within theuser interface; indicate the second feature within the second image; andprompt the user—through the user interface—to confirm that the secondfeature is analogous to the particular feature. If the user indicatesthat the second feature is incorrect, the system can repeat the featureselection routine to select an alternate feature from the second imageand repeat this process until the user indicates that the correctfeature was selected. The system can additionally or alternativelyprompt the user to manually indicate the correct feature in the secondimage, and the system can update the feature selection routineaccordingly. However, in response to receipt of manual confirmation ofthe second feature as analogous to the particular feature from the uservia the user interface, the system can: execute the feature selectionroutine at a third image in the set to identify a thirdfeature—analogous to the particular feature—in the third image; andexecute the feature selection routine at other images in the set toidentify analogous features in these other images.

3.6 Measurement Propagation

Block S344 of the third method S300 recites: extracting a first realdimension of the particular feature in the first assembly unit from thefirst image according to the measurement type; and, for each image in aset of images, extracting a real dimension of the feature in theassembly unit from the image according to the measurement type.Generally, once like features are identified in each image in the set ofimages in Blocks S310 and S340, the system extracts dimensions of eachof these features directly from their corresponding images.

Furthermore, once a dimension of a feature is extracted from an image ofan assembly unit, the system can render an indication of the feature andits dimension within the user interface, such as over the image oradjacent the images. In particular, in response to selection of a firstfeature from a first image at the user interface, the system can displaya first real dimension of the first feature with (e.g., on or adjacent)the first image within the user interface; in response to selection of asecond image of a second assembly unit at the user interface, the systemcan display both the second image and a second real dimension of asecond feature—analogous to the first feature—within the user interface;etc.

3.6.1 Real Dimension from Original Image

In one variation, the system: projects a dimension space onto the first(“flattened”) image; extracts the first real dimension of the particularfeature from the first image based on a position of the particularfeature relative to the dimension space and the measurement type; andrepeats this process for other images in the set.

In this implementation, the system can flatten an original digitalphotographic image of the first assembly unit and present the flattenedfirst image to the user through the user interface for selection of theparticular feature. Once the particular feature is selected, the systemcan project the particular feature from the flattened first image ontothe first digital photographic image to identify the particular featurein the original digital photographic image. The system can then map adistorted measurement space onto the first digital photographic image inpreparation for extracting a real dimension of the particular featurefrom the digital photographic image. Generally, in this variation, inorder to precisely (i.e., accurately and repeatably) calculate a realdimension of a feature of an assembly unit represented in a flattenedimage, the system can project a distorted measurement space onto thecorresponding digital photographic image and extract a dimension of thefeature from the digital photographic image based on a position of thefeature relative to the distorted measurement space. In particular,rather than extract a real dimension from a flattened image, which mayresult in loss of data over the original digital photographic image, thesystem can map a distorted measurement space onto the correspondingdigital photographic image in order to compensate for optical distortion(e.g., perspective distortion) in the digital photographic image whilealso preserving data contained in the image.

In one implementation, the system generates a virtual measurement spacerepresenting a plane in real space at a particular distance from acamera in the optical inspection station that recoded the digitalphotographic image but “warped” (i.e., “distorted”) in two or threedimensions to represent optical distortion in the digital photographicimage resulting from optics in the camera. In one example, aftercapturing a digital photographic image of an assembly unit, an opticalinspection station can tag the digital photographic image with a zoomlevel, focus position, aperture, ISO, and/or other imaging parametersimplemented by the optical inspection station at the instant the digitalphotographic image was recorded. In this example, to calculate adimension of a feature in the digital photographic image, the systemcan: extract these imaging parameters from metadata stored within thedigital photographic image; calculate a reference plane on which thereal feature of the assembly unit occurs in real space relative to thereal reference (e.g., a fiducial on the optical inspection station); andthen generate a virtual measurement space containing a set of X and Ygrid curves offset by a virtual distance corresponding to a known realdistance on the real reference plane based on the imaging parametersstored with the digital photographic image, as shown in FIG. 6. Thesystem can then calculate the length, width, radius, etc. of the featureshown in the digital photographic image by interpolating between X and Ygrid curves in the virtual measurement space overlaid on the digitalphotographic image.

In the foregoing example, the system can select a pixel or a cluster ofpixels at each end of a feature—analogous to the particular feature—inthe digital photographic image, project the pixels onto the warpedmeasurement layer, interpolate the real position of each projected pixelor pixel cluster based on its position relative to X and Y grid curvesin the measurement space, and then calculate the real length of thefeature (or distance between two features) in the real unit based on thedifference between the interpolated real positions of the pixels or acluster of pixels. In this example, the system can select a pixel or acluster of pixels at each end of a feature—corresponding to a featuredefined in the measurement specification—in the digital photographicimage, generate a virtual curve—representing a real straight line in themeasurement space and passing through the pixels or a cluster of pixels,and then calculate a straightness of the feature in the assembly unitfrom variations between pixels corresponding to the feature in thedigital photographic image and the virtual curve in the measurementspace. The system can therefore generate a warped measurement layer froma standard calibration grid, based on calibration fiducials in a digitalphotographic image, or based on any other generic, optical inspectionstation-specific imaging, or digital photographic image-specificparameter.

However, the system can generate a measurement layer (ormulti-dimensional measurement space) of any other form in Block S344 andcan apply this measurement layer to a digital photographic image in anyother way in Block S344 to calculate a real dimension of a feature onthe assembly unit according to parameters defined in the measurementspecification. The system can also render a virtual form of themeasurement layer, such as in the form of a warped grid overlay, over acorresponding digital photographic image when rendered with the userinterface.

3.6.2 Real Dimension from Dewarped Image

In another implementation, the system extracts real dimensions directlyfrom flattened images (described above). For example, when calculatingan homography transform for flattening digital photographic imagesrecorded at an optical inspection station, such as based on a referenceimage recorded at the optical inspection station, the system cancalculate a scalar coefficient that relates a length of a digital pixelin a flattened image to a real dimension (i.e., a length value in realspace), as described above. To extract a real dimension of a featurefrom an image in Block S344, the system can: count a number of pixelsspanning the feature; and multiply this number of pixels by the scalarcoefficient to calculate a real dimension of this feature. However, thesystem can implement any other methods or techniques to extract a realdimension of a real feature on an assembly unit from a flattened imageof the assembly unit. The system can implement these methods andtechniques for each image in the set of images to calculate realdimensions of like features across the set.

(The system can additionally or alternatively implement methods andtechniques described herein to calculate a dimension of an assembly unitsolely in one image of an assembly unit based on a measurementspecification (e.g., rather than propagate the measurement specificationacross all or a subset of images). For example, the system can implementthese methods and techniques to calculate a one-time measurement basedon a pixel-to-pixel selection entered by a user onto a single image.)

3.7 Access and Analytics

Block S350 of the third method S300 recites aggregating the first realdimension and a set of real dimensions extracted from the set of imagesinto a digital container (e.g., a virtual visual representation ordigital file). Generally, in Block S350, the system aggregates realdimensions of like (e.g., analogous) features extracted from the set ofimages—representing a set of assembly units—into a visual or statistical(e.g., numerical) representation of variations in this feature acrossthe set of assembly units.

3.7.1 Graphs and Charts

In one implementation, the system can compile real measurement values—oflike features in assembly units represented across the set ofimages—into a graphical plot. For example, the system can aggregate theset of real dimensions into a virtual histogram including a set ofdiscrete percentile ranges (e.g., 0-10%, 10-20%, 20-30%, etc.) spanningthe set of real dimensions and then render the virtual histogram withinthe user interface, as shown in FIG. 9A. In this example, in response toselection of (or placement of a cursor over) a particular percentilerange, in the set of discrete percentile ranges, the system can retrievean exemplary image—in the set of images—representative of the particularpercentile range and render the exemplary image over or next to thevirtual histogram within the user interface.

In another implementation, the system can: transform the set of realdimensions into a plot of real dimension versus time (e.g., time anddigital photographic image of an assembly unit as recorded along anassembly line) or versus serial number, as shown in FIG. 9B; and thendisplay this plot within the user interface. The system can alsocalculate a trendline or line of best-fit for this plot and predictdeviations from the nominal dimension of the feature—beyond thedimensional tolerance—based on this trendline. For example, the systemcan execute the foregoing methods and techniques substantially inreal-time as assembly units are assembled along an assembly line and asdigital photographic images of these units are received from an opticalinspection station arranged along the assembly line. In this example,the system can: generate a plot of real dimensions of the particularfeature—across the series of assembly units—versus serial number basedon data extracted from these images; and (re)calculate a trendline forthe plot following receipt of each additional digital photographic imagefrom the optical inspection station. If the trendline exhibits apositive slope (per unit time or per assembly unit) exceeding athreshold slope (e.g., once a threshold number of images of assemblyunits have been processed), the system can automatically generate a flagfor the assembly line and prompt the user (or other engineer or entityaffiliated with the assembly line) to review processes related to thecorresponding feature at the assembly line in order to preempt deviationof this particular feature from the assigned dimension beyond thedimensional tolerance. The system can therefore extract dimensions of aparticular feature from images, extrapolate trends in dimensions of thisparticular feature, and selectively prompt the user or other entity toinvestigate an assembly line in (near) real-time in order to achieveimproved yield from the assembly line.

In the foregoing implementation, the system can interface with the userthrough the user interface to configure a measurement specification foran assembly unit type or assembly line before any assembly units areassembled or imaged at the assembly line. For example, the system can:retrieve a three-dimensional CAD model of an assembly unit, as describedabove in the second method S200; display a two- or three-dimensional CADimage of the CAD model within the user interface in Block S320; receivea user selection of the particular feature directly from a CAD image inBlock S310; and configure a measurement specification for the particularfeature in Block S330 based on this selection and other data (e.g.,nominal dimension and dimensional tolerance of the particular feature,feature type, assembly type and configuration, etc.) extracted from theCAD model. The system can then extract a real dimension of afeature—corresponding to the particular feature selected from the CADmodel—from an assembly unit represented in each image received from anoptical inspection station on the assembly in (near) real-time accordingto this CAD-based measurement specification.

Alternatively, the system can retroactively apply a measurementspecification to a corpus of images previously recorded and stored in adatabase, such as to enable the user to access dimensional statistics ofa particular feature in a batch of assembly units from a previousproduct model when designing a next product model. For example, thesystem can extract an average dimension, a standard deviation, deviationfrom a nominal dimension, instances of deviation from the nominaldimension beyond a preset dimensional tolerance, or any other statisticrelated to real dimensions of a like feature across a set of (similar)assembly units represented in a set of images. The system can thenpresent these graphical and/or numerical data to the user, through theuser interface. However, the system can generate any other type of plot,graph, or chart of dimensions of the particular feature across the setof assembly units or extract any other statistic from these dimensionaldata.

Alternatively, the system can package these dimensions into aspreadsheet or other digital file or database; the user may thendownload this file for manipulation of these dimensional data withinanother program. However, the system can package these dimensional datainto any other digital container in Block S350.

3.7.2 Outliers

In another implementation, the system detects dimensional outliers andflags individual assembly units accordingly, such as in real-time asdigital photographic images are received from an optical inspectionstation or asynchronously based on images stored in a database. Forexample, in the implementation described above in which the systemgenerates a histogram of dimensions of a like feature across the set ofimages, the system can: identify a subset of images representingfeatures exhibiting dimensions falling within an upper bound (e.g., top10%) of dimensions represented in the histogram; and flag serialnumbers—stored in image metadata or extracted directly from visual datacontained in these images—of these assembly units, as shown in FIG. 9A.Similarly, the system can: compare dimensions of like features extractedfrom the set of images directly to a nominal dimension and a dimensionaltolerance associated with the feature, such as via manual entry or byextracting these data from a CAD model or engineering drawings, asdescribed above; and flag serial numbers of assembly units containingthis feature exhibiting dimensions that differ from the nominaldimension by more that the dimensional tolerance, as shown in FIG. 9B.

Therefore, in this implementation, the system can: access a targetdimension of the particular feature and access a dimensional toleranceof the target dimension of the particular feature, such as by promptingthe user to manually enter these values when configuring the measurementspecification or by extracting these values from a CAD model orengineering drawings. The system can then: flag a serial number of asecond assembly unit including a second feature—analogous to theparticular feature—characterized by a real dimension differing from thetarget dimension by more than the dimensional tolerance.

The system can execute this process in (near) real-time as images arereceived from an optical inspection station deployed along an assemblyline. In the foregoing example, the system can: configure themeasurement specification at a first image based selection of theparticular feature in the first assembly unit; identify the secondfeature—analogous to the first feature—in the second assembly unitrepresented in the second image recorded at the assembly line at asecond time succeeding the first time; extract a second real dimensionof the second feature in the second assembly unit from the second imageaccording to the measurement specification; and, at approximately thesecond time, transmit an electronic notification—containing a the fourthassembly unit—through the user interface or to another computing deviceassociated with the user.

In this implementation, the system can also access a range of dimensionsof a feature—analogous to the particular feature selected in BlockS310—associated with failure of an assembly, such as stored in adatabase or manually entered by the user during the current imageviewing session. Upon receipt of a second image of a second assemblyunit, the system can: identify a second assembly unit containing asecond feature—analogous to the particular feature—and characterized bya second real dimension contained within the range of dimensionsassociated with assembly failure; and then serve a prompt to inspect thesecond assembly unit to an electronic account associated with a user,such as in (near) real-time as the second assembly unit passes throughthe assembly line. The system can implement similar methods andtechniques to asynchronously flag an assembly unit containing a featureof dimension within this failure range. For example, if testing of asubset of assembled assembly units following production of a largerbatch of assembly units later indicates a correlation between adimension of a particular feature and failure, the system can flag otherunits in this larger batch that were untested for assembly units thatmay fail due to analogous features exhibiting dimensions within thefailure range.

3.7.3 Multiple Real Dimensions

In one implementation shown in FIG. 9C, the system implements Blocks ofthe third method S300 described above to apply a first measurementspecification and a second measurement specification—distinct from thefirst measurement specification—to the set of images and then compilesresults of the first and second measurement specifications into onegraphic or statistic.

In this implementation, the system can: receive manual selection of asecond particular feature in the first assembly unit represented in thefirst image in Block S320; receive selection of a second measurementtype for the second particular feature in Block S330; and extract asecond real dimension of the second particular feature in the firstassembly unit from the first image according to the second measurementtype in Block S344. In this example, the system can then, for each imagein the set of images: identify a second feature in an assembly unitrepresented in the image in Block S340, wherein the second feature inthe assembly unit is analogous to the second particular feature in thefirst assembly unit; and extract a second real dimension of the secondfeature in the assembly unit from the image according to the secondmeasurement type in Block S344. In Block S350, the system can thenpopulate a two-dimensional graph with points representing the firstmeasurement type across the set of images (e.g., along a first axis ofthe graph) and representing the second measurement type across the setof images (e.g., along a second axis of the graph). For example, thesystem can define a Cartesian coordinate for each image in the set as(result first measurement specification, result second measurementspecification) and then represent each image in a two-dimensionalscatterplot according to its Cartesian coordinate.

In this implementation, the system can also receive a mathematical modellinking the first measurement specification to the second measurementspecification, such as: a maximum and/or minimum difference between aresult of the first measurement specification and the second measurementspecification; a nominal sum and summary tolerance for a sum of a resultof the first measurement specification and a result of the secondmeasurement specification; etc. for one assembly unit in the set. Thesystem can then implement methods and techniques described above toautomatically flag an assembly unit containing features that violateeither individual nominal dimensions and dimensional tolerances for thefirst and second measurement specifications or the model linking thefirst and second measurement specifications.

3.7.4 Filters

In one implementation, the system: receives a set of filter values andselection of a preexisting measurement specification from a user;filters a set of images captured by optical inspection stations in anassembly line according to the filter values; automatically applies themeasurement specification to the filtered set of images, as describedabove, to generate a graphical plot and/or numeric representation of thedimension of the feature across the corresponding set of units; and tocommunicate the graphical plot and/or numeric representation to theuser. For example, the user can search through or filter a set ofpreexisting measurement specifications configured for an assembly,build, or configuration based on an assembly line identifier, opticalinspection station serial number, measurement specification origin orowner (e.g., user who originally configured the measurementspecification), measurement specification name, feature name or type,etc. and select a particular measurement specification (or a set ofmeasurement specifications) to apply to a set of images captured by anoptical inspection station inserted along an assembly line. In thisexample, the user can also enter one or more unit filters, such as build(e.g., EVT, DVT, PVT), configuration (e.g., color, vendor, engineeringdesign), assembly date (or date range, time range), assembly stage, unitserial number range, timestamp, optical inspection station or assemblystate, fixture serial number, measurement value, measurement range, etc.The system can also store these filters, group units based on any of theforegoing parameters, and generate new plots from a set of imagescorresponding to a set or subset of images based on one or more filtersselected by a user.

In the foregoing example, the user can enter textual, natural languagefilters into a prompt window, such as within a native SMS messagingapplication executing on a smartphone, within a native validationtesting application executing on a tablet, or within a browser window ona desktop computer, and the system can implement natural languageprocessing techniques to transform a textual string entered by the userinto a set of filters for units—and to a select a corresponding set ofimages and/or other data—for application of the measurementspecification. Alternatively, the system can issue dynamic dropdownmenus for filters available for a set of units previously imaged basedon a measurement specification and/or other filters selected by theuser. The system can then generate a graphical plot of dimensions of thefeature across the filtered set of units, as described above, and pushthis graphical plot back to the user, such as in the form of a staticfigure viewable in the native SMS text messaging application executingon the user's smartphone or in the form of an interactive graph viewablein the native validation testing application or within the browserwindow.

However, the system can interface with the user through the userinterface executing at a local computing device in any other way toreceive a measurement specification and/or assembly unit filters. Thesystem can then: filter the set of images according to one or morefilter values selected by the user to select a subset of imagesrepresenting a subset of assembly units; generate a graphical, textual,or statistical representation of like features—analogous to theparticular feature—contained in the subset of assembly units; and thenserve this graphical, textual, or statistical representation to the userthrough the user interface or other native application or web browserexecuting on the user's computing device.

3.7.5 Notifications

In the implementation described above in which the system executes theforegoing methods and techniques in (near) real-time, the system canserve a prompt to inspect a flagged assembly unit to an electronicaccount associated with the user in real-time. For example, the systemcan render an inspection prompt containing an inspection prompt and aserial number of a flagged assembly unit directly through the userinterface. For the system that executes Blocks of the third method S300remotely from the user interface, the system can additionally oralternatively generate an SMS text message or an application-basednotification containing an inspection prompt and a serial number of aflagged assembly unit and then transmit this text message ornotification to a mobile computing device (e.g., a smartphone, asmartwatch) associated with the user, such as in (near) real-time whenthe user is occupying a building housing an assembly line or the flaggedassembly unit is currently in production or is currently housed.Alternatively, the system can serve graphical, textual, and/orstatistical representations of dimensions of like features across theset of assembly units asynchronously, such as when the user activatesthe measurement specification within the user interface or by sendingdaily or weekly digests of results of the measurement specification tothe user via email, as described below.

In one variation, the third method S300 can also include: receiving asubscription to the measurement specification from a user, such as inthe form of a request to receive updates relating to dimensionsextracted from images of new assembly units according to the measurementspace; and distributing the digital container to an electronic accountassociated with the user based on the subscription. In this variation,the system can enable the user—and other users—to subscribe to themeasurement specification and can automatically push graphical and/ortextual prompts—such as described above—to computing devices associatedwith each user subscribed to the measurement specification. For example,the system can push an electronic notification—to inspect a particularassembly unit containing a feature analogous to the particular featureand exhibiting a dimension deviating from a nominal dimension definedfor the particular feature or representing a statistical outlier (i.e.,not necessarily outside a predefined tolerance)—to a mobile computingdevice associated with each user subscribing to the measurementtechnique in real-time. Alternatively, the system can populate aspreadsheet containing serial numbers of assembly units containingfeatures exhibiting such deviation from the nominal dimension, insertthe spreadsheet into an image, and send the email to subscribers of themeasurement specification, such as in a daily or weekly digest.

3.8 Image Review

The system can also execute the second method S200 in conjunction withthe third method S300 to virtually align images by a common featurecontained in assembly units represented in these images and to extractdimensions of the common or other like feature contained in theseassembly units. In this implementation, the system can then define anorder of images in the set based on dimensions of like featuresextracted from images in the set and then scroll through the set ofimages at the user interface according to this order, thereby enablingthe user to view images of assembly units in order of increasing (ordecreasing) dimension of a common feature contained in these assemblyunits and visually aligned by this common feature. For example, thesystem can: define an order of the set of images based on realdimensions of features, analogous to the particular feature, extractedfrom images in the set of images; virtually align images in the set ofimages by features analogous to the particular feature (e.g., by acorner feature or by two edge features), as described above; and, inresponse to a scroll input at the user interface, index throughrendering images in the set of images within the user interfaceaccording to the order.

In another implementation, the system can implement Block S252 of thesecond method S200 in conjunction with Block S350 of the third methodS300 to select two (or more) images representative of a span ofdimensions of a feature—analogous to the particular feature—and togenerate a composite image containing these representative images. Forexample, in Block S350, the system can: calculate a range of realdimensions spanning the set of real dimensions extracted from the set ofimages; select a second image—from the set of images—representing asecond assembly unit containing a second feature analogous to theparticular feature and characterized by a second dimension proximal afirst end of this range of real dimensions; select a third image—fromthe set of images—representing a third assembly unit containing a thirdfeature analogous to the particular feature and characterized by a thirddimension proximal the opposite end of the range of real dimensions;generate a composite image including the second image and the thirdimage overlayed over the second image, as in Block S252; and then renderthe composite image within the user interface. The system can thenimplement methods and techniques described above to combine two or moreimages representing a subset of assembly units containing like featuresrepresentative of the set of assembly units as a whole based on realdimensions of these features extracted from the set of images.

The systems and methods described herein can be embodied and/orimplemented at least in part as a machine configured to receive acomputer-readable medium storing computer-readable instructions. Theinstructions can be executed by computer-executable componentsintegrated with the application, applet, host, server, network, website,communication service, communication interface,hardware/firmware/software elements of a user computer or mobile device,wristband, smartphone, or any suitable combination thereof. Othersystems and methods of the embodiment can be embodied and/or implementedat least in part as a machine configured to receive a computer-readablemedium storing computer-readable instructions. The instructions can beexecuted by computer-executable components integrated bycomputer-executable components integrated with apparatuses and networksof the type described above. The computer-readable medium can be storedon any suitable computer readable media such as RAMs, ROMs, flashmemory, EEPROMs, optical devices (CD or DVD), hard drives, floppydrives, or any suitable device. The computer-executable component can bea processor but any suitable dedicated hardware device can(alternatively or additionally) execute the instructions.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the embodiments of the invention without departing fromthe scope of this invention as defined in the following claims.

We claim:
 1. A method comprising: displaying a first image of a firstassembly unit within a user interface, a form of the first imagerecorded at an optical inspection station; in response to a change in aview window of the first image at the user interface: displaying a firstsubregion of the first image within the user interface; identifying aset of discrete surfaces on the first assembly unit represented withinthe first subregion of the first image; selecting a first discretesurface exhibiting a greatest dimension in the first set of discretesurfaces; identifying a first feature that bounds the first discretesurface; and locating a first virtual origin on the first feature on thefirst assembly unit represented in the first image; recording a geometryand a position of the first subregion of the first image relative to thefirst virtual origin; in response to receipt of a command to advancefrom the first image to a second image at the user interface: locating asecond virtual origin at a second feature on a second assembly unitrepresented in the second image, the second feature on the secondassembly unit analogous to the first feature on the first assembly unit;projecting the geometry and the position of the first subregion of thefirst image onto the second image according to the second virtual originto define a second subregion of the second image; and displaying thesecond subregion of the second image within the user interface.
 2. Themethod of claim 1, wherein displaying the first image of the firstassembly unit comprises: retrieving a first digital photographic imagefrom a database, the first digital photographic image recorded by theoptical inspection station at a first time during an assembly period;normalizing the first digital photographic image based on a referenceimage recorded at the optical inspection station to generate the firstimage; and serving the first image to a computing device executing theuser interface for rendering.
 3. The method of claim 2, furthercomprising: retrieving a second digital photographic image from thedatabase, the second digital photographic image recorded by the opticalinspection station at a second time during the assembly period; andnormalizing the second digital photographic image based on the referenceimage to generate the second image.
 4. The method of claim 1, whereinlocating the second virtual origin at the second feature on the secondassembly unit represented in the second image comprises: projecting aboundary encompassing and offset from the first virtual origin onto thesecond image; identifying a set of edge features on the second assemblyunit represented within a region of the second image contained withinthe boundary; and identifying the second feature, in the set of edgefeatures, exhibiting a second geometry analogous to a first geometry ofthe first feature; and locating the second virtual origin on the secondfeature.
 5. The method of claim 1, wherein locating the first virtualorigin at the first feature on the first assembly unit represented inthe first image comprises locating the first virtual origin, of thefirst image, on the first feature defining a first corner of a firstpart in the first assembly unit; and wherein locating the second virtualorigin at the second feature on the second assembly unit represented inthe second image comprises: identifying a set of parts in the secondassembly represented in the second image; selecting a second part, inthe set of parts, analogous to the first part; and locating the secondvirtual origin, of the second image, on the second feature defining asecond corner of the second part.
 6. The method of claim 1: whereindisplaying the first subregion of the first image within the userinterface comprises displaying the first subregion of the first imagewithin the user interface in response to the change in the view windowof the first image at a first time; wherein identifying the firstfeature that bounds the first discrete surface comprises: identifying aset of edge features on the first assembly unit within the firstsubregion of the first image; at a second time succeeding the firsttime, receiving selection of a pixel within the first subregion of thefirst image; and selecting the first feature, from the set of edgefeatures, proximal the pixel.
 7. The method of claim 1, whereindisplaying the first subregion of the first image within the userinterface comprises displaying the first subregion of the first imagewithin the user interface in response to the change in the view windowof the first image comprising a zoom input over the first image.
 8. Themethod of claim 7: wherein recording the geometry of the first subregionof the first image relative to the first virtual origin comprisesrecording a zoom level selected for the first subregion of the firstimage; wherein recording the position of the first subregion of thefirst image relative to the first virtual origin comprises recording avertical offset and a horizontal offset between the first virtual originin the first image and an origin of the first subregion; and whereinprojecting the geometry and the position of the first subregion of thefirst image onto the second image according to the second virtual originto define the second subregion of the second image comprises: defining ageometry of a second view window according to the zoom level; verticallyoffsetting an origin of the second view window from the second virtualorigin within the second image according to the vertical offset;horizontally offsetting the origin of the second view window from thesecond virtual origin within the second image according to thehorizontal offset; and defining a region of the second image bound bythe second view window as the second subregion.
 9. The method of claim8: wherein locating the first virtual origin on the first feature on thefirst assembly unit represented in the first image further comprises:locating the first virtual origin, of a first coordinate system, on thefirst feature within the first image; and aligning a first axis of thefirst coordinate system to the first feature; wherein recording thegeometry and the position of the first subregion of the first imagerelative to the first virtual origin further comprises: recording anangular offset between an edge of the first subregion of the first imageand the first axis of the first coordinate system; and wherein locatingthe second virtual origin in the second image at the second feature onthe second assembly unit represented in the second image comprises:locating the second virtual origin, of a second coordinate system, onthe second feature within the second image; and aligning a second axisof the second coordinate system to the second feature; and whereinprojecting the geometry and the position of the first subregion of thefirst image onto the second image according to the second virtual originto define the second subregion of the second image further comprises:angularly offsetting an edge of the second subregion from the secondaxis of the second coordinate system according to the angular offset.10. The method of claim 1, further comprising: serving a prompt toconfirm a projected geometry and a projected position of the secondsubregion of the second image through the user interface; in response toreceipt of confirmation of the projected geometry and the projectedposition of the second subregion of the second image: retrieving a setof images of assembly units from a database; locating a virtual originin each image in the set of images; and projecting the geometry and theposition of the first subregion of the first image onto each image inthe set of images to define a set of subregions for the set of images;and indexing through the set of subregions displayed within the userinterface in response to a scroll input at the user interface.
 11. Themethod of claim 1, further comprising: setting a first opacity of thefirst subregion of the first image; setting a second opacity of thesecond subregion of the second image; overlaying the second subregionover the first subregion to generate a composite image; displaying thecomposite image within the user interface; adjusting the first opacityaccording to an input received at the user interface; adjusting thesecond opacity as an inverse function of the first opacity; andrefreshing the composite image in response to the input.
 12. The methodof claim 1, further comprising: retrieving a virtual computer-aideddrafting model representing the first assembly unit; generating acomputer-aided drafting image of the virtual computer-aided draftingmodel at an orientation and in a perspective approximating anorientation and a position of the first assembly unit represented in thefirst image; locating a third virtual origin at a third feature in thecomputer-aided drafting image, the third feature analogous to the firstfeature on the first assembly unit; projecting the geometry and theposition of the first subregion of the first image onto the virtualcomputer-aided drafting model according to the third virtual origin todefine a third image; and displaying a translucent form of the thirdimage over the first subregion of the first image within the userinterface.
 13. The method of claim 1: further comprising, at a secondtime succeeding the first time, receiving selection of a pixel withinthe first subregion of the first image by a user via the user interface;and wherein identifying the set of discrete surfaces on the firstassembly unit comprises identifying the set of discrete surfaces on thefirst assembly unit represented proximal the pixel within the firstsubregion of the first image.
 14. A method comprising: displaying afirst image of an assembly unit in a first stage of assembly within auser interface, the first image recorded at a first optical inspectionstation; locating a first virtual origin in the first image at a firstfeature on the assembly unit represented in the first image; in responseto receipt of a zoom input at the user interface, displaying a firstsubregion of the first image within the user interface; storing ageometry and a position of the first subregion of the first imagerelative to the first virtual origin; identifying the first feature onthe assembly unit in a second image of the assembly unit in a secondstage of assembly; locating a second virtual origin in the second imageaccording to the first feature; defining a second subregion of thesecond image based on the geometry and the position of the firstsubregion of the first image and the second virtual origin; locating athird virtual origin in the second image at a second feature on theassembly unit represented in the second image; identifying the secondfeature on the assembly unit in a third image of the assembly unit in athird stage of assembly; locating a fourth virtual origin in the thirdimage according to the second feature; defining a third subregion of thethird image based on the geometry and the position of the secondsubregion of the second image and the fourth virtual origin; in responseto receipt of a command to advance from the first image to the secondimage at the user interface, displaying the second subregion of thesecond image within the user interface; in response to receipt of acommand to view the first image, the second image, and the third imageconcurrently at the user interface: setting a first opacity of the firstsubregion of the first image; setting a second opacity of the secondsubregion of the second image; merging the first subregion, the secondsubregion, and third subregion into a composite image; and displayingthe composite image within the user interface.
 15. The method of claim14: wherein displaying the first image of the first assembly unitcomprises: retrieving a first digital photographic image from adatabase, the first digital photographic image recorded by the firstoptical inspection station arranged at a first position along anassembly line; normalizing the first digital photographic image based ona first reference image recorded at the first optical inspection stationto generate the first image; and serving the first image to a computingdevice executing the user interface for displaying within the userinterface; and wherein displaying the second subregion of the secondimage within the user interface comprises: retrieving a second digitalphotographic image from the database, the second digital photographicimage recorded by a second optical inspection station arranged at asecond position along the assembly line; and normalizing the seconddigital photographic image based on a second reference image recorded atthe second optical inspection station to generate the second image. 16.A method comprising: displaying a first image of a first assembly unitwithin a user interface, the first image recorded at an opticalinspection station; in response to a change in a view window of thefirst image at the user interface: displaying a first subregion of thefirst image within the user interface; identifying a set of discretesurfaces on the first assembly unit represented within the firstsubregion of the first image; selecting a first discrete surfaceexhibiting a greatest dimension in the first set of discrete surfaces;identifying a first feature that bounds the first discrete surface; andlocating a first virtual origin on the first feature on the firstassembly unit represented in the first image; recording a geometry and aposition of the first subregion of the first image relative to the firstvirtual origin; identifying a second feature represented in a secondimage of a second assembly unit, the second feature analogous to thefirst feature; projecting the geometry and the position of the firstsubregion of the first image onto the second image according to thesecond feature to define a second subregion of the second image; and inresponse to receipt of a command to advance from the first image to thesecond image at the user interface, displaying the second subregion ofthe second image within the user interface in replacement of the firstimage.
 17. The method of claim 16: further comprising locating a firstvirtual origin, of the first image, on the first feature defining adatum on a fixture retaining the first assembly unit in the opticalinspection station; wherein recording the geometry and the position ofthe first subregion of the first image relative to the first featurecomprises recording the geometry and the position of the first subregionof the first image relative to the first virtual origin; whereinidentifying a second feature represented in a second image of a secondassembly unit comprises: identifying a set of features represented inthe second image; selecting the second feature, in the set of features,exhibiting dimensional characteristics and geometric characteristicsapproximating dimensional characteristics and geometric characteristicsof the first feature; and locating a second virtual origin of the secondimage on the second feature defining a second corner of the second part;and wherein projecting the geometry and the position of the firstsubregion of the first image onto the second image comprises projectingthe geometry and the position of the first subregion of the first imageonto the second image according to the second virtual origin to definethe second subregion of the second image.
 18. The method of claim 16,wherein recording the geometry and the position of the first subregionof the first image relative to the first feature represented in thefirst image comprises: projecting a boundary encompassing and offsetfrom the first feature onto the second image; identifying a set of edgefeatures represented within a region of the second image containedwithin the boundary; and identifying the second feature, in the set ofedge features, exhibiting a second geometry approximating a firstgeometry of the first feature.
 19. The method of claim 16: furthercomprising, at a second time succeeding the first time, receivingselection of a pixel within the first subregion of the first image by auser via the user interface; and wherein identifying the set of discretesurfaces on the first assembly unit comprises identifying the set ofdiscrete surfaces on the first assembly unit represented proximal thepixel within the first subregion of the first image.